Event Replay: Inside Applied AI Engineering at OpenAI

Allie Sandza Wood: [00:00:00] Hi everyone, and welcome back to the OpenAI Forum. I'm Allie Sandza Wood, and I'm part of the OpenAI Global Affairs team. One topic we return to often at the forum is the future of work. And today we're bringing that conversation inside OpenAI with a closer look at applied AI engineering, a fast-growing field that brings together advanced AI, engineering, and real-world customer needs. We'll begin with an introduction from Bonnie Chatterjee, followed by a conversation with applied AI engineers Erika Kettleson and Thomas Li about their roles and what they've learned. We'll leave time at the end for your questions, so please submit them throughout the session. With that, I will hand it over to Bonnie.

Bonnie Chatterjee: [00:00:48] Thank you, Allie. Hi everyone. Our mission is to ensure that AGI benefits all of humanity. The Applied AI engineering team contributes to that mission by helping people to put that technology to work. We are a customer-facing engineering team that closes the gap between what our models can do and a system that our customers, like yourselves, can use, trust, and gain value from. We sit within the go-to-market organization and focus on three things. Being experts on our own products, helping our customers build impactful products and solutions, and then bringing that field insight back into research and product. So today, we will share an up-close and personal view of what this work actually entails, how we strive for excellence, and the passion and magic behind it all. To do that, I would like to introduce my panelists Thomas Li, applied AI engineer for the enterprise, and Erika Kettleson, applied AI engineer working on startups. So to kick this off, Erika, let me ask you this question. What does a representative week look like, and which part of the work requires the most judgment?

Erika Kettleson: [00:02:00] Well, I think something that everyone on this team appreciates is there's no representative week, but if you work back from our goals, I think there's a few commonalities, and that's customer work first and foremost, so talking to customers in Slack, meeting builders and engineers in person, hosting them in OpenAI in our offices, figuring out what they're working on, what problems they're experiencing. That's the number one. Every week is going to have a lot of those, and I think that's the most fun, joyful part of the work. And then what's in service of that is the learning and the building. So if I want to help my customers, I need to know our product best, so I'm going to build with our APIs, I'm going to figure out where the edges of capabilities are. I want to be an expert in prompting models. I want to discover problems before they even come up. And then there's that kind of communicating back piece, which is, how do I talk to our research, our product and our engineering teams about what I'm seeing in the field, and then kind of connect those dots. So tons of meetings, both with customers and then internal facing teams, and then big bursts of building. And I think that is why our team is unique, because you get both of those. Deep building and engineering work, and then also super fun, chatty, communicative customer work.

Bonnie Chatterjee: [00:03:24] Yeah. That sounds great. And I think, Tom, would you have a perspective from the enterprise lens?

Thomas Li: [00:03:28] Yeah, I think there are a lot of similarities there. I would say in enterprise, as you can imagine, enterprises are very complex organizations definitely, so there's a lot of potentially multiple teams working on the same product, interdependencies, different priorities, and timelines. So there's a portion of the role that's how do you help customers navigate that complexity. But then also, I absolutely agree with Erika. The work really ebbs and flows. I think there is a baseline of I have regular sessions with customers, making sure we're staying on top of progress, checking in with them, unblocking any issues they're running into. And then there's also short-term sprints where we're really working on things like, hey, we're going live in a month. We need to work on evals, we need to drive latency and cost optimization. And those end up being a bit more intense. So in terms of judgment and where I spend my time, it's also just figuring out what's the highest leverage use of my time. Is it unblocking a process thing or a people problem, or is it really just digging deep into the technical work?

Bonnie Chatterjee: [00:04:27] Yeah. That's a good point. And so let me take this back. So you talked a little bit about bursts and you talked a little bit about models. How do you keep ahead of all the things we're doing, given how fast we launch and how fast we're shipping?

Thomas Li: [00:04:41] Yeah. That's a tough one, I would say. I struggle with that. Even just with OpenAI's products itself, it's quite a lot to learn, let alone the broader industry and how rapidly it changes. I would say for me there's a couple things that really matter. One is just having that curiosity and that drive.

Thomas Li: [00:04:58] So, you know, as new features are released, like, you know, having that mentality of, I'm going to go dogfood this, I'm going to go experiment, get hands-on and, you know, play around with the tooling, try to figure out how it works, how it might apply for my customers. But then also there's, you know, the element of, like, we have very strong enablement teams internally that can help coach us. And really, I think it's important not to, you know, expect to be an expert on everything, but rather you need the breadth to understand everything that's out there and how it might relate to your customer. But then specifically, like, I might go deep on certain areas where it matters for my customer or, like, I might lean on, you know, folks like Erika when I have a specific topic area where maybe I'm not the expert, but I know my colleagues are.

Bonnie Chatterjee: [00:05:41] Yeah. And to follow up on that, let me ask you this question. So, we talked a little bit about models and, you know, learning the rapid speed of that. How do we collaborate with, let's say, research or product who are doing a lot of the shipping and what we need to learn them in order to take them to our customers?

Erika Kettleson: [00:06:03] I think it's interesting because we sit at this place where we're seeing a ton of patterns on the customer side. So we know what's on our customers' roadmaps, what do they want to build, and then we know what's on our roadmap. And we can let, one, where is research excelling right now and how can we bring that? And then also, yes, how can we inform them? But we're kind of pattern matchers. So we need to understand what problems are actually low hanging fruit, but maybe our research and engineering teams don't have this customer point of view to understand, oh, this small change would be a huge impact. So we're kind of matching that. What is something that customers want that we should place bets on? Like, hey, this is a big deal for... We need to completely redo post-training or whatever it may be, but we know that customers are really going to care. Versus with something that's actually pretty feasible. So I think we act as this pattern matching, this kind of communication linchpin between these two worlds. And I think that's why engineering appreciates us. Because we have that perspective from customers. And customers trust us. They tell us what's going on and vice versa.

Bonnie Chatterjee: [00:07:12] Yeah. And I think it's important for us to bring back what we're learning from our customers back to our products and research. And our teams are so excited to hear from our customers. Yeah. And we take the feedback so seriously. It's one of the things I am the proudest of at being at OpenAI is how much all of our teams care about what the customers have to say. That being said, though, every once in a while we run into challenges. So let me ask you, Tom, a non-confidential example, please. Maybe a challenge you ran into and how did you solve it?

Thomas Li: [00:07:45] Yeah, definitely. So I would say one big one top of mind for me is we've been working with a large retailer recently on how do we look at optimizing their shopping agent that they have deployed on their website. And specifically about driving latency down to sub-five seconds. And really that helps unlock a bunch of new experiences and product surfaces where they can embed this solution. Now the challenge for us initially was we had the code, the harness, and some of the evals, but we can't quite replicate the exact environment that they're running in. We don't have internal APIs that we can rely on to sort of mirror and test those evals. So for us it was a lot of how do I recreate and mirror or replicate their harness so that I can get a proxy for what my customer is seeing and really do that experimentation, try to hill climb on evals and be able to share those results back with my customer. So that was what we worked on. So a core portion of that was figuring out how do we mock out specific synthetic responses from APIs, feed that back, and then from there, once we had that harness, we were able to do a lot more benchmarking across a number of different models and reasoning efforts to really narrow in on what are the opportunities here at play for the customer that we could really enable.

Bonnie Chatterjee: [00:09:04] Yeah. It's valid. I mean, there's a lot of demand for us to go build these evals and help with these benchmarks. Is that something you're seeing also, Erika?

Erika Kettleson: [00:09:12] Yeah, absolutely. I mean, as you're talking, I'm kind of thinking a big part of our job is building trust internally and externally, so getting access to systems, getting access to data, creating representative evals for our teams to use, absolutely, and there can be variances across enterprises and startups of how willing people are to share certain data or evals or problems, but across our whole organization, that is probably core to our success, is coming with the technical chops and the curiosity that you want to make them succeed, such that they trust you, because that is what we want. At the end of the day, we want to see people launch things into prod they're really excited about and scale them and just be able to make products that they couldn't without us.

Erika Kettleson: [00:09:56] That's the most fun part. Yeah, I think that building trust piece is sort of what you're speaking to, and that's regardless of what customer you're working with.

Bonnie Chatterjee: [00:10:05] Yeah.

Bonnie Chatterjee: [00:10:06] Yeah, I would agree. I think building trust and ensuring that what we're doing is always at the service of the customers, and helping them move their impact, helping them with their value, I feel like those are core to what we do. Another question we get a lot is, what does it take for someone to thrive in an environment like this? We have a crazy shipping schedule, we have insane hours of work, and there's a lot of excitement that goes into what we're doing. But from my perspective, and I'd love to hear both of your feedback on this, but from my perspective, I think some of the core characteristics is curiosity, like an honesty of pursuit, like doing it for the sake of getting to the best results possible for the customers. And then high agency, when we see something we go fix it, we all feel super empowered to go just fix the thing. And so I'd love to hear from you, and maybe Erika first, and then Tom, what do you think would make someone successful in our environment?

Erika Kettleson: [00:11:06] Yeah, I'm being a little crazy. You know, definitely curiosity, kind of just being obsessed with the technology. Like you get in here, you open Slack, you're like, I can't even, I just have access to the coolest knowledge in the entire world, no one's going to stop me, I can just learn everything, like it's amazing. So having that kind of, like, allowing yourself to follow your curiosity, and then learn how to work with the people around you as well. Like Tom was saying, like, we can't learn it all. So knowing how do you build relationships across the organization as it grows, where you know, like, oh, I can go to Bonnie for this, and Tom does a lot about how retail customers are building this, and kind of giving back, and then knowing how to like collaborate, because you can't do it all. So there's sort of like curiosity for yourself, like you kind of want to like get all this knowledge for yourself. But also, like how, who can I go to that can distill their knowledge for me? Yeah, so I think those kind of, yeah, that really, and being an office is so great for that. And yeah. But like, I think our teams are really good at this knowledge sharing, like gathering up all the knowledge and then sharing back. And the generosity to share, but I feel like every one of us is so excited to help others that it never feels like a chore it always feels like, Oh, I get to say X to Y.

Thomas Li: [00:12:22] Yeah, I think for me, the one that sticks out is, it's just a sense of ownership and really like, you know, driving to, you know, the best results for your customers. So I think like, you know, as we've mentioned in this role, you have a lot of trust, a lot of autonomy to sort of do what you think is right. But what that also means is there's no one looking over your shoulder to make sure you're doing exactly like every single thing that you need to do. So it's sort of on you to, you know, champion, you know, and really represent the interests of your customer, you know, follow up on threads of things that are, you know, relevant to them, maybe alphas that they're interested in participating, and sort of seeing that through and making sure you don't drop the ball. So to me, like, you know, somebody who has that persistence to continue, like making sure they're following through. That that's huge.

Bonnie Chatterjee: [00:13:08] Yeah, I love it. So you've been talking about culture. Let's take this one more question. I'm going to start with you, Tom. What would you say is distinctive about our culture at OpenAI?

Thomas Li: [00:13:19] Yeah, so the one I like most is, you know, I personally feel like everybody in our org is really a believer of our mission. As we said, like ensuring AGI benefits all of humanity. And I think from what I've seen on our team, like the way that manifests is, you know, helping customers figure out where does AI actually fit in their, you know, products in their, in their companies, whether that's like improving efficiency of how they operate internally, whether it's about creating like better user experiences for their own employees or for their end users, right. But really being able to see that come to life and sort of reach more people, I think is something that really energizes all of us. So that's, that's something I love.

Bonnie Chatterjee: [00:14:00] So, yeah, reaching all the people, that's important. And for us, the lens we look at it through is work, we help a lot of people, we are helping our customers, they have, like you said, their own employees, their own end users. Erika, what would you say is distinctive about our culture?

Erika Kettleson: [00:14:16] Well, I think we were talking about this yesterday. I love this idea of urgency and agency. Because we have this strong sense of urgency, and we can match the urgency of our customers. Right? It's like we are right there with you. Same thing internally, we're working on a project, let's sprint. But we have the agency to do it. Yeah. Like I said, like you get in Slack, you're looking at our repos, you're reading what our engineering teams are up to. So you have this agency to understand the full landscape. Right? You're allowed to play with all the tools. You have unlimited tokens. Okay, all the tokens you want. Go crazy with Codex. There are no limits. But those things combined mean like you can just really, you can really build stuff. You can do whatever you want. You can talk to anybody.

Erika Kettleson: [00:14:54] You can build stuff, you can do whatever you want, you can talk to anybody, make connections, no one's gonna hold you back except for your own time limitations, with being a human, I don't know.

Thomas Li: [00:15:05] We're not agents yet?

Erika Kettleson: [00:15:06] No, no, no, not yet, not yet.

Bonnie Chatterjee: [00:15:09] I would add to this that the thing I find very distinctive is the humility. I have had the opportunity to work with some of the smartest people I've ever met, including the two of you, and I find that the way we show up for each other, for our customers with this humility at the center of our ethos, I find that very fulfilling and I think we all thrive in this atmosphere because we have all of these little areas, the urgency, the agency, the ability to help a lot of people, but we do it with a sense of humility and I love that about us.

Erika Kettleson: [00:15:47] Totally, yeah.

Thomas Li: [00:15:49] Day one, come, we just wanna see what you wanna build.

Erika Kettleson: [00:15:51] No, that's what we're here for.

Bonnie Chatterjee: [00:15:54] All right, with that, let me pass it back to Allie who's gonna take some questions from you all, our audience.

Allie Sandza Wood: [00:16:00] Thank you and I totally agree with those things you said about our culture. It's really wonderful. Our first question comes from, and I'm sorry if I mispronounce this name, Saptarshi Bhattacharjee, a software development engineer at Amazon and they ask, what is the difference between software engineering and applied AI engineering orgs at OpenAI? How do the two orgs work together and also how to both coordinate with the research teams?

Bonnie Chatterjee: [00:16:27] Great question. Erika, do you wanna go first and I can add some after?

Erika Kettleson: [00:16:31] Sure. I think you can think about us as specifically customer-facing. So we're dedicated to specific engineers and builders who are building our API platform using Codex and ChatGPT. Whereas internal platform engineers, like our applied team, they're focused on building OpenAI products and building our APIs. So it's like we're customer-facing. I think that's like the fundamental difference between the two organizations. And we kind of work with both them and research depending on kind of what our customers' projects are. But I think the fundamental difference is really just the customer-facing. I would say that. And I think to the second part of the question was how do we work with product and research. From our perspective, it's really bringing the customer's voice, their feedback, their insights back to both product and research. Whereas our internal software engineer teams, again, are building the product themselves. So the way they would interact with research would look a little bit different than ours.

Thomas Li: [00:17:31] Absolutely. Yeah, I mean, echo that entirely, Bonnie. And I think with these sort of tools with AI, there's really such an unbounded amount of things you can do. So being there with the customer and sort of seeing all the interesting, fascinating things they're doing, that's a lot of signal that our product teams just don't get to see otherwise. We probably have a little bit wider scope too, because when you're working with a customer, you're thinking broadly across all of their problems whereas usually non-applied AI engineers will have a little bit less scope. Where you're thinking, how can I help every part of their business regardless of what model or API? Almost like a breadth versus a depth thing where our internal engineers are deeply caring about the thing they're building and we are caring about helping a broader range of customers.

Allie Sandza Wood: [00:18:18] Okay, great. This next question comes from Daniel Greene. How do you keep up with the speed of new AI launches and capabilities? Does your team ever sleep? That's the question.

Bonnie Chatterjee: [00:18:29] No. Short answer, no. But I think Tom, you shared a little bit earlier, so maybe Erika, you tell us. How do you learn new things? Do you ever? Have you ever? Erika, please.

Erika Kettleson: [00:18:42] It is a marathon, not a sprint. Really, because when people ask me about working here, I always say, no one will ever tell you to slow down or stop. You have to decide when it's enough. So I really, and I believe that, because it's like, this is so exciting. I love this job, but there are, yeah, I have to set it. But how do I learn? I'm in Codex constantly. I'm building stuff. I think building, talking to people who have already built something or learned something, and then just getting in Codex, reading Slack, getting really deep into just getting your hands on the technology, every single day, you have to do that. And you have to place your bets. You cannot do everything. There are things I just have blinders on for. I'm like, I'm gonna ask someone about that, but I can't focus on that right now. We have too many launches a week.

Bonnie Chatterjee: [00:19:29] Yeah, and I think we also do a really good job of learning from each other. And then our product and research teams also do a good job of telling us the things that are key. So we do tend to get a lot of inbound information, and then it is up to us to prioritize what we're learning.

Allie Sandza Wood: [00:19:47] Okay, great. This next question comes from Nene Ndukwe. As you're serving the customers, are you also consolidating?

Allie Sandza Wood: [00:19:52] Serving the customers, are you also consolidating your learnings into larger patterns that can be built back into what OpenAI builds or produces? Or productizes, is the other word mentioned.

Thomas Li: [00:20:03] So there is some piece on productizing that maybe Bonnie can speak to after. But I'll say from my end, right now I largely support the retail vertical. So for us, a lot of this now is going out to the field, working with various retailers, understanding sort of common patterns and solutions that we see these retailers building. And figuring out what are the common primitives to that? How do we build maybe reusable assets, whether it's a solution pattern or architecture diagrams, that we can use to go and accelerate other customers doing similar types of work? So a lot of this is definitely pattern recognition, and figuring out how do you consolidate that information, build an asset that you can then bring to other customers and try to improve.

Bonnie Chatterjee: [00:20:43] And we have many different forums by which we bring the customers' feedback and their sort of needs and insights back into product that does then get prioritized. So I think it's a very self-fulfilling loop. The more we talk to customers, the more they trust us, the more they tell us, the more we can bring back, and the better our products get. So yeah, I think that's a very integral way of how we behave and how we function.

Allie Sandza Wood: [00:21:12] Okay, great. Another question, how have tools like Codex changed how you approach your work?

Bonnie Chatterjee: [00:21:19] Sure, a lot. You wanna go first, Erika?

Erika Kettleson: [00:21:22] Well, I can start because when I started at OpenAI, there was no Codex. GPT-4o was the model of choice. So my job has changed a lot since then, as I know everyone everywhere has. But it's just made it so that doing research and building are so much faster. And you can just have more ideas and then actually just shoot them off, come back, let them run. You're learning new things. You're learning to work with Codex as opposed to, I'm really good at searching whatever, Slack or something. You're learning these new tools. But you can just do so much more and research in a way that is so much deeper. It's incredible. And run little side experiments. Because everyone's along the journey with us. Our data analysis team means that we can use Codex to then do incredible data analysis that we wouldn't have been able to do. So it empowers every team, which then we get to, rising tides lift all ships. We really get to benefit from everyone at OpenAI using Codex. It's not just me. It's a very universal thing here.

Thomas Li: [00:22:29] Yeah, I would say on my end, it's, yeah, the same when I joined, it was like 4o, I think. And it's been very dramatic. I would say the biggest thing is, I think in the past, you would be a lot more single-threaded on a specific issue. You'd look at a document and work it to completion. I think now there's almost like a shift in mentality where it's like, I need to figure out how to fire off all the tasks I want, try to parallel process everything, and go breadth first, and then think about that delegation aspect of how I work. And so that's really changed a lot of how I almost approach problems that I wouldn't have thought about a year ago. We're all managers now. Of our fleet of agents. I would say the other interesting thing that you touched upon, Erika, is that everyone else uses Codex at OpenAI. And because all of our functions are on it, it makes it that much easier to collaborate in a way. Or if we need something from a different department, it's just that much easier because a lot of their work is also available. And so, it really has changed how we work and think and function.

Allie Sandza Wood: [00:23:33] Agreed all around. Our next question is from Joe Todaro.

Allie Sandza Wood: [00:23:39] Yeah, Todaro. How does Applied AI Engineering partner with GTM Strategy and Operations on complex customer deployments, and where does that partnership add the most value?

Bonnie Chatterjee: [00:23:49] That's a great question. So our strategy and ops teams in go-to-market are a very valuable part of our org where they bring insights about some of the patterns that Tom was talking about, but from a more abstracted way. So in terms of what kinds of deployments are getting what kinds of results and what segment of our population should we focus on more? And so that overarching information is really helpful for us as we are trying to prioritize our limited resources always towards all of these opportunities in front of us and how we can make the best of what OpenAI can bring to our customers. So I think it's a very tight partnership. We both sit in go-to-market, so that also makes it a little bit easier for us to really closely collaborate. But we tend to essentially work together with our customers.

Allie Sandza Wood: [00:24:44] OK, great, and this will be our last audience question. This comes from Victor Fuentes. Given that applied AI engineering...

Allie Sandza Wood: [00:24:50] Given that applied AI engineers and FDEs both work with customers in production AI, how do the roles differ in practice and what does a typical applied AI team look like?

Bonnie Chatterjee: [00:25:01] Great question. Also, so primarily our FDE team works on scope projects. A lot of times sort of figured out milestones and contracts ahead of time. A lot of the work they do does sometimes come back into product as something we will then productize over several customers that they've worked with. So I think the nature of that engagement tends to look a little bit different. For the applied AI engineering team, we work with a wider breadth of customers and the work we do tends to be more sort of parallel play, like hand-in-hand with our customers. So rather than building a lot of things for them, we are often building things with them and sort of helping them learn as we go. So that's, I think, a key difference in how we approach how we help our customers. But again, the goal of all of our teams is very similar, which is make our customers successful, help them gain the value from our intelligence that we get to bring to them, and then just bring them value that they can then propagate into the world themselves.

Allie Sandza Wood: [00:26:09] Well said.

Bonnie Chatterjee: [00:26:10] Thank you.

Bonnie Chatterjee: [00:26:11] Thank you, Allie. Back to you.

Allie Sandza Wood: [00:26:14] Of course. Thank you all so much. Thank you, Erika, Bonnie, and Tom. And thank you, of course, to everyone who joined us today to learn more about Applied AI Engineering at OpenAI. It's a great conversation.

Allie Sandza Wood: [00:26:25] For people tuning in, I hope everyone will join us for another OpenAI Forum event on September 3rd. We will have a livestream of the Economic Opportunity Demo Day with OpenAI and the GitLab Foundation, and we will share the registration link in the chat. So until then, thank you all so much for joining and have a great day.

Bonnie Chatterjee: [00:26:45] Thank you.