Event Replay: Inside OpenAI: How OpenAI Teams use Codex to Do More

[00:00:00] Hi everyone, thank you so much for joining us. Today's discussion is focused on how agents are transforming work, particularly work that is outside of coding. It should be a great conversation. As you may have seen, on Thursday we launched ChatGPT Work, which brings Codex's agentic capabilities beyond coding to work across apps and files. Codex remains our dedicated coding agent, but both now live in the ChatGPT desktop app. We're hoping that today gives you ideas about how agents can impact your work no matter what field you work in.

[00:00:43] So we're going to start today with OpenAI's Chief Economist, Ronnie Chatterji, and Drew Johnston, a member of the technical staff on our ECON team. They're going to discuss research into how agents are transforming work inside and outside of OpenAI, and then we're going to have teams from across our company share short demos of how agents are changing their day-to-day workflows. And then, of course, everyone will be back to answer your questions. So for now, I'm going to hand it over to Ronnie. Ronnie, take it away.

[00:01:13] Ronnie: Thanks so much, Allie, and great to be here with Drew Johnston from the ECON research team. Welcome to OpenAI Forum.

[00:01:22] Drew: Hey, Ronnie. Great to be here. I figured I'd start people off just with the origin story. As you know, every great session needs an origin story. Why did we write this paper? What's so interesting about codecs at this point in time, and where agentic AI is going?

[00:01:40] Ronnie: Yeah. So this paper was in part inspired by my own experiences. So I came on board at OpenAI back in January, so six months ago, for anyone keeping score at home, which already feels like it was a lifetime ago. Switching jobs is always a pretty big change. You have to go and configure your healthcare plan, set up your 401k, all of that. But this was a much bigger change than I was used to. At the same time as I was dealing with all the normal, like administrative stuff, I was also kind of managing a transition in what I did day to day in my job, which was very much brought about by the fact that I was moving from an environment in which agentic AI was not common into one in which it very much was.

[00:02:30] So coming onto this job, the composition of what I did changed pretty much overnight. I went from writing and testing all of my own code by hand in my last job to working at OpenAI where agentic tools can help with all of that automatically. In the past couple of months, I haven't written any code by hand and agentic tools are also really useful for setting up tests and helping me to find the data that I need for the analyses I was doing. So working on this for a couple of smaller projects I'd worked on previously, like really made it very clear to me that this had pretty profound implications. This paper was inspired by trying to understand how these technologies rolled out at OpenAI and how they're rolling out in the economy more broadly.

[00:03:17] Drew: Yeah, Ronnie, I feel what's special about this as researchers is we're almost like participant observers as they used to do in the old days in like anthropology, where we're actually going through this revolution at work and our own team. As you know, you've been on the vanguard within our team in terms of getting us all to use Codex in new and exciting ways. We're also studying how Codex is impacting the economy at large. And so a lot of the inspiration from our research comes from the use cases that we're encountering every day. And that's what makes this paper so exciting. It also goes down to how we choose the methods and the analysis plan.

[00:03:51] Ronnie: And I want you to talk a little bit about what I think my favorite part of the paper is — the different samples that we study Codex adoption because it's not just consumers or enterprise. It actually has a special twist. I think people are going to really appreciate it. So tell people about the samples and why we constructed them that way.

[00:04:08] Drew: Yeah. So in the paper, one thing that we do that is novel is we're able to contrast how people are using Codex both inside OpenAI but also outside of it as well. And we're able to look at two pretty distinct groups of users outside of OpenAI. So the first is organizational users. So you can think of people whose account is supplied for them by someone else. Like, perhaps your business gets you a license to use the tooling. And then also people who have individual accounts. So that'll be if you have a free plan or if you purchased a pro plan for yourself.

[00:04:41] So we're able to look at how Codex usage across those three account types differs. And there's pretty profound differences across the three. So, yeah, we have a graph in the paper that shows the extent to which usage has diffused across these three groups.

[00:04:58] Speaker 1: Across these three groups.

[00:05:00] And you can see that within OpenAI, pretty much everybody is already using agentic tooling. More than 95% of people at the company in the past month have used Codex. That includes both the people who are working on research and engineering here, as well as people who are on the legal teams and on HR and on comms.

[00:05:19] So within OpenAI, people have really made the transition from the past generation of conversational AI tools to making use of agentic tools that are much more capable. We see that in other populations, this transition is still a lot earlier.

[00:05:30] I will caveat that we pulled this data. The paper was finalized back in early June, which feels like a lifetime ago already, but then only about like one in five or one in six people with an organizational account was using Codex in a given month. Most people are still interacting with AI through traditional conversational harnesses.

[00:05:50] Among people who had individual accounts, this was even less widespread; less than 1% of people were making use of Codex and agentic technologies. So the environment inside OpenAI was really looking quite different. The two bottom series were rising quite rapidly, but there's still pretty limited uptake in the broader world of agentic tooling.

[00:06:18] Our paper was also interested in the reasons for why this might be the case. So within OpenAI, obviously a lot is different than outside of it. There's pretty widespread enthusiasm for these tools within the company. A lot of within teams, there's a lot of champions who have kind of been promoting usage and sharing knowledge about workflows that can be enabled by agentic tooling.

[00:06:41] There's leadership encouragement for experimentation and using these tools as widely as possible. There's more exposure to usage with like a really dense community of people who know what they're doing and are familiar with these technologies. There's a lot of spillovers that just occur when you're talking to your coworkers about how to best use these tools to speed up workflows.

[00:07:05] Speaker 2: Yeah, Drew, when I look at a chart like this, I always think about the fact that we're sort of living in the future. If you look at that OpenAI trendline, but it's always a question as a researcher, to what extent is this a look into the future or maybe an organization that's very novel and idiosyncratic.

[00:07:20] What we're seeing, and we saw this today with the 7 million milestone, is we're continuing to gain users. What's especially exciting is to watch how that ramp is going to happen in other enterprises.

[00:07:32] One of the things I think is so interesting, it was the other chart that really went viral on Twitter for us, is this idea of the non-technical roles, or I should say the less technical roles, because we're also included in that. This chart here shows that really well. Research and engineering were pretty saturated, let's say by the end of last year, but quickly, and maybe this is a Drew Johnson effect on the chart, right? Joining in January, you see a huge uptick in all these other functions, highlighting finance where we sit, recruiting, legal.

[00:08:02] This kind of closing the gap, I think is remarkable. It's just a quarter basically. What do you make of this? What's happening as we go from technical to less technical? Do we expect to see this kind of trend in other kinds of firms? And what would be the contingencies in expecting that?

[00:08:18] Speaker 1: Yeah, so Codex initially was kind of like seen as a tool for coding. So you could use it to read and write and test all the code files on your computer in a way that is much more seamless than was possible with conversational tools.

[00:08:37] I think what we've seen is that people have realized how impactful this can be outside of the coding domain in which it's traditionally been used most widely.

[00:08:48] One thing that also is pretty striking from the chart is that within fields that were later adopting, such as recruiting or legal, you've seen that convergence has been quite a bit faster. The slope of those lines in April is pretty sharp there, where kind of in the span of two or three weeks entire departments switched from using mostly conversational tooling to mostly agentic tooling quite rapidly.

[00:09:14] I think this kind of speaks to the fact that the functionality in a lot of use cases has been there for a long time. And it really, once that was in place, it really just takes a champion and it takes willingness and it takes a team being willing to rethink some of its workflows and to formalize some of the kind of tacit knowledge that is floating around.

[00:09:37] Once that's in place, things can transition pretty rapidly and teams can begin making use of these new workflows that are opened up quite rapidly.

[00:09:47] Speaker 2: Well said, Drew. And when I see this, we've talked about a capability gap for a long time when it comes to AI; it's clear from the benchmarks that AI capabilities are advancing really fast. And the big question is...

[00:09:56] really fast and the big question is, [00:09:58] why aren't more people using them in that way? [00:10:00] And what you see here at OpenAI is our own version [00:10:03] of that capability gap closing. I mean, you think of the revolution we're living through [00:10:07] from chat bots to reasoning and now to agentic work [00:10:10] where it's not just asking for an answer, it's delegating work. [00:10:13] You're seeing that gap close very quickly at OpenAI. [00:10:16] And I think what's interesting is to watch some of the other companies going through this [00:10:18] and seeing the enterprises adapt their workflows to these new tools. [00:10:24] I do feel your comment about sort of the near past, [00:10:27] June 11th, I think is when the sample ended. [00:10:29] And I joked today on social media, I said, it's like we're studying economic history now. [00:10:34] People want the most up-to-date data, normally up to June would be pretty darn good [00:10:38] as we see here in July, but for people who are thinking about where agentic AI is today, [00:10:42] it seems like ages ago already. So I think that's what I'm excited about [00:10:46] to keep drawing this chart, seeing how it looks like in other enterprises. [00:10:49] And that's why the foundation we laid in the paper is so exciting for me.

[00:10:53] I wanted to ask you a little bit about those non-technical folks or less technical folks. [00:10:57] And we were included in that sample. You know, I can think about a world where, [00:11:01] you know, agentic AI comes at me and I start using to do the things [00:11:04] I was already doing at work. I can think about a world where I start to do new things [00:11:08] that maybe weren't part of my job. Those are two different kinds of forces [00:11:12] with non-technical or people adopting codecs. [00:11:16] How are they actually using it in their work, at least according to our study, [00:11:19] which is the first of its kind. So it gives us some indication of what to expect.

[00:11:24] Yeah, so as part of the blog post we released alongside this paper, [00:11:28] we put out some statistics about people's job titles and within each job title, [00:11:32] the types of tasks that are most prevalent within codecs usage. [00:11:39] So we see that there's a lot of developers that are using codecs today [00:11:41] and software engineering is still the biggest workflow overall and related things such as, [00:11:45] you know, development environment management and things of that nature. [00:11:52] We do see some instances in which people are using codecs [00:11:57] in ways that blur the boundaries between different types of job types [00:12:01] that we're used to seeing historically. [00:12:03] So we've seen some pretty interesting examples of this within OpenAI [00:12:07] where people are using agentic tooling to enable themselves to do things [00:12:11] that previously would have required large-scale collaborations across the company [00:12:15] or things that were totally outside of their skillset.

[00:12:21] So there was a pretty cool example on our team just last week. [00:12:24] So our team right now, we put out a call for applications [00:12:30] for what we're calling the OpenAI Research Exchange, [00:12:33] which is a program that lets people from the outside, [00:12:36] external researchers, come in-house to work on a project at OpenAI that they can propose. [00:12:42] So we put out this call for proposals and we got something like 500 applications for it, [00:12:47] which is amazing and always what you want. [00:12:51] But that's a pretty intimidating thing for our team to review. [00:12:55] All of these applications got dropped in one enormous spreadsheet. [00:13:00] And reading the applications, some of which were several pages long, [00:13:04] in one cell on a spreadsheet was pretty painful for our team.

[00:13:10] So our chief of staff, who is amazing and super capable, [00:13:13] but not a technical person and who has no programming experience, [00:13:19] was able to use codecs to take that spreadsheet [00:13:21] and turn it into an interactive website [00:13:23] that presents all the details from the proposals in line, [00:13:27] has links to all the documents that people who are proposing projects attached, [00:13:34] and allows people to, in a web form, submit their comments on it, [00:13:36] give it ratings on all of the metrics that we assign, [00:13:40] and have those populated back into the spreadsheet.

[00:13:43] So this is coming from someone whose official title is Chief of Staff [00:13:46] and who has no programming experience. [00:13:49] So I feel pretty comfortable saying that like, [00:13:52] some of the normal job boundaries are being blurred here, [00:13:55] and also that a lot of work that would not have gotten done previously is getting done. [00:14:00] In the context of most jobs, if the Chief of Staff was given the spreadsheet, [00:14:05] we would have made it work with the spreadsheet, [00:14:06] it would have been kind of painful and not very user-friendly, [00:14:10] but in the end, it would have been fine. [00:14:12] Here, we were able to speed up our review process a ton [00:14:14] and do it as a much more friendly way using this website, [00:14:21] the creation of which would not have been possible without Codex. [00:14:25] I mean, seeing this underneath the hood, [00:14:27] and we went through it together, Drew, for this one, [00:14:29] it's that lump of labor fallacy that you and I have often talked about, [00:14:31] it's like there isn't an upper limit on the amount of work you can do. [00:14:34] So what happened here, we got this amazing number of applications, [00:14:39] we wanna give a careful review, this is really important, [00:14:41] we love working with outside researchers to try to answer the biggest questions in AI, [00:14:46] how do you give each one a good review? [00:14:47] A lot of it is just form factor, [00:14:49] if it's presented in a spreadsheet like it was, very difficult. [00:14:52] She was able to build this interactive website very quickly.

[00:14:54] Speaker 1: websites very quickly, and then all of a sudden you can run through those applications in a much more efficient manner, people get better reviews, the team actually moves with pace, which is really important. And we end up being able to do things that we couldn't do before. And so when she was able to add that skill, right, website development to her task list, I think it's a good window into how AI's gonna be used by a lot of people, basically expand their agency in terms of what they can do and help us do more as a team.

[00:15:19] So I felt the same way when I was interacting with that site. How do you think about this when you think about the workflows within agentic capability?

[00:15:28] Speaker 2: I think one of the most interesting things about agents is the ability to do parallelization, being able to do multiple things at once, and also the ability to build repeated scalable tasks and automations, and we'll see some of this later with our skills feature. How do those kind of angles play into the study we did and what you found?

[00:15:49] I'd be interested to hear, you think about some of these advanced skills that people are putting to work with agentic AI.

[00:15:53] Speaker 1: Yeah, so one of the things we were super interested in in the paper is how people are using these new technologies to change their workflows. And I think the concurrent agents part of it is one of the most interesting aspects.

[00:16:07] So for context here in CodeX, you can have multiple threads running simultaneously, doing work on different things in a way that keeps the context to each contained within it and lets you work on things in a way that doesn't interfere across projects if that's what you wanna do.

[00:16:23] So this is something that people kind of understand intuitively. We see that among consumers and among organizational users, a pretty good share of users in a given week will have multiple threads running concurrently at some point.

[00:16:37] One thing that surprised me coming into this paper is the extent to which within OpenAI this has been supercharged. So within OpenAI in the given week, something like 10% of users peak at having more than 10 agents running simultaneously at some point.

[00:16:52] And this has been a really powerful workflow for me in that it lets me kick off an agent to do some data analysis and simultaneously kick off another agent to clean up some messy bits in the code base. And another one can be checking a paper I drafted for typos simultaneously.

[00:17:10] And on paper, I'm an individual contributor at OpenAI. I don't manage anyone, but a lot of the time that I spend here, I spend feeling like I'm a manager in some capacity. And I think a lot of people who on paper are listed as individual contributors in their job, if they're using agentic AI to its fullest capacity, will often feel the same way in which thinking like a manager can be a pretty valuable skill in dividing up your workflows across these threads and across these agents who can be doing a bunch of work for you at the same time.

[00:17:44] Speaker 2: Yeah, do you have another agent who's sending me World Cup scores? Cause I feel like I get some slapstick about that. But you can do that too. But go on to your other point.

[00:17:52] Parallel workflow is one really important thing. You see like more than a quarter, I think, in our study of people who are sort of spitting up more than five or more agents, I think, from the paper. You can tell me the exact stat, but that's pretty interesting as an indication of work.

[00:18:05] Tell us a little bit more about some of the other advanced things like skills and maybe also like the horizon of the task. I found that to be an interesting result from the paper too.

[00:18:12] Speaker 1: Yeah, so one thing that we've been very interested in is the ways in which users are using skills within CodeX to kind of codify workflows that their team has. So we've been interested in the ways in which people can share knowledge within a team without having to do formal instruction.

[00:18:25] So it might be that your team has a very specific process that you use to format the weekly rollup of your team's activity before sending it to your executive sponsor. Previously, if you were gonna bring someone new onto the team, you would probably have to have someone senior sit down with them and teach them like, hey, we always bold all the subheadings here and we always format them in exactly this way.

[00:19:01] Skills kind of give you a way to give repeatable, shareable instructions to CodeX about how you want a specific workflow to be performed. And by using skills, you can document these workflows kind of in something that can just be shared freely among members of your team.

[00:19:14] And that will make the work automatically done in a certain way. We think this is a very interesting process in that it can really reduce the barriers to coming on board and to picking up new workflows for a given person.

[00:19:26] We found like a huge increase since the beginning of the year in usage of skills, both within OpenAI, but also externally among individual and organizational users. And we think this is something that matters a lot for understanding how teams are collaborating within them and how teams are formalizing various workflows that they do.

[00:19:49] So we thought that was very interesting as well.

[00:19:52] And I think, Drew, what I'm picking up from this and the results you take them together, first, we're moving to an era where instead of just asking questions, you can delegate work and then you can delegate work in multiple streams at a time as you're doing often and leading the way on our team in many ways in that.

[00:20:07] And then you can delegate increasingly longer tasks. We're seeing some things in the data of people delegating tasks in the hours and a percentage of really lead users who are doing more than a workday in terms of adding up all the work that their agents are doing.

[00:20:20] And then once you do that, you can almost make them sort of routine every day tasks in a way that's really going to change the nature of work. So it's working in parallel and systematizing those. I really feel like that's going to be the new operating system for work in a lot of ways.

[00:20:34] And that is consistent with how you think about how general purpose technologies are diffused often into organizations. As, you know, something amazing is invented, we have to figure out exactly how to use it in the organization. And I see the early shoots of that, at least in our work.

[00:20:47] Let me ask you one final question. And then I'll turn it over to Melanie to move over to the next part of our session here.

[00:20:55] What's your favorite way to use Codex? People are dying to know out there. You wrote the paper with us. You're the lead author. What do you think is your favorite way to use Codex?

[00:21:03] Yeah. So actually, I use Codex to write all of our one-on-one.

[00:21:08] Oh, now you tell me, Drew.

[00:21:11] Okay.

[00:21:12] Yeah. So I have Codex in advance of our weekly meeting. I tell it like, hey, go scan my calendar, scan my Slack, go scan our task management tool, and summarize everything I've been working on for the past week.

[00:21:24] And I'll put that at the beginning of the doc. We can talk through everything that I've accomplished or should have accomplished in the past week. And then also scan my calendar for the coming week and summarize everything that I've got coming up, everything that's mentioned in my task management tool.

[00:21:36] This was something I tried doing by hand a few times and it was super painful, involved like looking across weeks worth of messages on a million chat threads across several applications, remembering everything I did. It was a mess.

[00:21:53] And having Codex be able to do it and take all those facts and create a one pager and send it to Rani automatically has been a huge boon to my productivity and has let me accomplish a lot more stuff that is more meaningful for me to work on than summarizing my accomplishments.

[00:22:11] And both of us, we have a more productive meeting because it's laid out there and then I can also analyze it using agenda tools like Codex so it works out really well and allows us more time to talk about what's important.

[00:22:20] Drew congrats on the paper. It's been awesome working with you on this. We'll be around to answer more questions from the audience, but before we do that, I want to hand it to my colleague, Melanie Avilbebe from our Data Science team to take you on the next step with Codex.

[00:22:32] Melanie.

[00:22:34] Hi, everyone. I'm Melanie. I am a data scientist at OpenAI. I've been here for a year and I'm currently working on how to make our agendic products better at data work.

[00:22:44] I'll go through a demo today of a core data workflow, which is a KPI metrics update. Like most data teams at OpenAI, we have important metrics that we review every week and we want to understand what changed, why it changed and what the team should do next.

[00:23:01] If you want to go ahead and play the video.

[00:23:05] So my basic approach with Codex is simple. I connected to as much of my working context as possible. Here you can see underneath the plugins tab I have a bunch of different plugins installed.

[00:23:15] If you want to just pause the video quickly right here. Plugins help connect Codex to the tools that we use every day at work. It allows it to retrieve information and take action on our behalf.

[00:23:26] I've installed our data warehouse, our business intelligence tool, our experimentation platform, as well as Google Drive and Slack. Once Codex has access to that working context, I don't need to start by deciding which tool to open or which query to write.

[00:23:39] I can start with the outcome I want and let Codex coordinate the work across those systems. Even in an AI assisted world, producing a trustworthy KPI update can take a data scientist one or even two days.

[00:23:53] It means querying data, investigating different metric movements, incorporating business contexts, and creating something that the team can ultimately review. So Codex lets me approach that as one end-to-end workflow, and we can see what that looks like for a KPI update if you want to go ahead and play the video.

[00:24:12] So here I've simply prompted Codex to produce this week's KPI update. This is deliberately an outcome level request. I'm asking Codex to coordinate the work flow, not merely help me write individual queries.

[00:24:25] As Codex works, we can discuss the process. So first I've used a custom skill called Northstar KPI Context, which you can see tagged in the prompt, that our team created for this workflow specifically.

[00:24:36] This skill captures the things that we shouldn't have to explain every week, like which metrics matter, how they're defined, where the trusted datasets live, which dimensions are useful for investigation, and how we want the finished update to be structured.

[00:24:50] You can think of the skill as reusable.

[00:24:50] You can think of the skill as reusable institutional knowledge for this workflow. [00:24:54] I'm also using the Data Analytics plugin also tagged in the prompt. [00:24:59] This plugin helps Codex perform the analytical work. [00:25:02] It basically serves as a recipe for how to produce a high-quality metrics update, [00:25:06] including how to validate data, compare week-over-week performance, [00:25:10] identify the largest contributors to movement, and creating the appropriate visualizations. [00:25:16] Because our experimentation and product contexts are connected as well, [00:25:20] Codex can actually go beyond saying the metric moves. [00:25:22] It can actually help answer the question the team really cares about, which is why. [00:25:27] In this example, it was able to use Databricks to query data, [00:25:31] and it was able to use our Google Drive connector to find experiment results [00:25:35] and incorporate that into the update, both of which you can see in the [00:25:38] upper right-hand corner source panel. If we just give this a moment, [00:25:42] it should produce the results in just a second. [00:25:44] Great. So, you can see a Slack summary in the response, [00:25:51] and then on the right-hand side, an Analytics report. [00:25:55] So, this update will lead with an executive summary and the current state of the business, [00:26:00] and then it shows the important movements and likely drivers associated with the analysis, [00:26:05] and it includes different charts and supporting evidence so that someone can [00:26:09] inspect the analysis and go deep rather than simply relying on a generated summary. [00:26:15] We can also just take a look at the report here. It has different sections including [00:26:20] recommended next steps and caveats and assumptions. You can also do cool things [00:26:24] like dive into the underlying data source so you can vet the data. [00:26:28] If you want to pause the video here before we move on. [00:26:32] So, this process isn't meant to put our business decisions on autopilot. [00:26:36] My role has shifted from assembling the update to now reviewing and extending the analysis, [00:26:42] so I'll check whether the data looks correct, whether the proposed explanation matches what [00:26:46] the team knows, and whether there's additional context or an investigative avenue that Codex [00:26:51] may have missed. So for example, in the report, I noticed that it flagged that European revenue [00:26:57] was down significantly. I can immediately ask it to investigate further. If you want to go ahead [00:27:02] and play the video, you can see that in action. So, Codex is going to continue the investigation [00:27:08] using the same context, data, and working artifacts that are already in this thread. [00:27:13] The first output isn't necessarily the end of my analysis, it just gives me a strong starting point [00:27:18] for asking better follow-up questions. And once this workflow is working well, [00:27:23] I can schedule it via an automation underneath the schedule tab. Every Thursday morning, [00:27:29] an automation will run this prompt against fresh data. So that when I start my day, [00:27:34] the first draft of the deck and the Slack update is ready for my review, [00:27:38] along with any anomalies or suggested investigations that might need my attention. [00:27:42] The automation will handle the repeatable work, and then I stay responsible for the interpretation [00:27:47] and what ultimately gets shared to the team. So that's an example of performing a KPI update [00:27:53] in CodeX. The last thing I want to just drive home before wrapping up is that the most valuable [00:27:58] change that CodeX has made is that it can take an entire recurring workflow, like a KPI update, [00:28:04] and turn it into a reusable system that can save me days of work, [00:28:07] which then gives me, the data scientist, more time to investigate the questions [00:28:11] that actually require human judgment. Thank you, everyone. I'll now pass it over [00:28:16] to Blaine to share a design demo.

[00:28:25] Hi there. I'm Blaine. I'm a UX designer here at OpenAI. Designers here, we use CodeX every day to get [00:28:29] ideas into a tangible form faster. We add polish directly into the production application [00:28:35] and submit pull requests to update the product. We collect and synthesize feedback [00:28:40] on autopilot with scheduled tasks, and we generate Sigma Mocs in bulk. [00:28:45] We also recently released a plugin for CodeX called Product Design that has some of the [00:28:50] utilities we've been using internally to ideate, explore, and prototype even more directly in [00:28:56] CodeX. So let me show you what I mean. Whenever I have a new idea, I'll start by just going [00:29:02] straight to CodeX and asking it to make it more concrete. So here I've got a vague idea for a [00:29:08] meetings or calendar experience I want to add to chat GPT, but I don't know what I want, [00:29:12] so I'm asking CodeX to explore the first pass for me. [00:29:16] Now, under the hood, we've done a lot of work to give CodeX all of our design system canon [00:29:21] and even use CodeX to create screenshots of every flow in production. This gives CodeX [00:29:26] a really good sense of what makes chat GPT look and feel like chat GPT. [00:29:32] It's generated a few rough mocks for us here, and you can see it looks like it's part of the [00:29:37] chat GPT product already. We've got a timeline concept, a more standard calendar concept, [00:29:43] and an inboxy style concept. These are really cool, [00:29:46] and what I love about this workflow is it kind of inspires me to think about.

[00:29:48] is it kind of inspires me to think about other ideas that I want. So I can ask Codex to mock up more things based on whatever this sparks in my mind. I asked it to try another concept and again it's going to go through this process of just ideating and coming up with some rough mocks for what that might look like. This one looks pretty good. It's just a really simple view and we can ask it once we're happy with it to prototype it into something interactive and fully fleshed out.

[00:30:17] Here behind the scenes again it's going to be using all of the source material from our production code base and from our Figma design system to generate a working prototype that is going to look and feel a whole lot like the ChatGPT that we already know. Without me having to do any of the hard work. So after a few minutes of cooking on that we can open it up directly in Codex and we've got an interactive prototype that looks and feels kind of like a ChatGPT product.

[00:30:34] Now it's obviously still rough and this is actually my favorite part because once we get something going, I can get into this flow of making changes, critiquing it, reviewing it, and brainstorming. I like to turn on this annotation feature and just kind of go wild. Add all of my thoughts, ideas, and critiques directly to the page and maybe I want that to be a single line. Maybe I want to change the button layout here etc. etc. And I can just fire all of these off into a queue to Codex and it'll start fixing these in real time.

[00:31:22] Once we send it off, I can just kind of watch while my prototype improves before my eyes. So here you'll see the visual design gets improved. Some of the layout gets improved and I can then start that process all over again. I can find new things that I want to change or new ideas that I have and add those comments to the queue for Codex to work on when it's done with what I've already requested.

[00:31:38] What I love about this is this actually gets me into the actual work of designing of like what I want it to do and what might feel good and I don't have to worry about finding the right Figma base mock or I don't have to look up anything about our design system or find the right icon. It's doing all of that rote work on my behalf so I can just focus on getting a delightful and fun experience to explore my new idea.

[00:32:01] And once we've done this, I can kind of keep this going as long as I want and get it really polished. I can add animations and transitions just by asking it what to do. I don't have to move to another proprietary product to prototype anything and I have the capabilities of anything that code can do at my fingertips.

[00:32:24] Once I'm done with this and it's in a place where I'm happy to share with my team, Codex makes it really easy to share in all kinds of ways. I can obviously publish this website and share it out to my team and this prototype is fully operational and working. But I can also ask Codex to make a movie for me or make an animated GIF to share in Slack if that's a better approach.

[00:32:46] And in this way I can get a lot of feedback really quickly on ideas without having to worry about the rote aspects of managing our design system. So I hope that was a fun look at how we do some design in Codex here at OpenAI and I'll pass it over to Ken to talk about sales.

[00:33:11] Hey if you can actually pause the video. Okay great hey I'm Ken and I help lead our SCR team here in the go to market organization and I use Codex I guess all the time especially as a leader because our team's quite large I lead scaled efforts here so there's a lot that typically I would need like a dedicated data analyst or a systems person for and then like an ops person for but have for the last you know year or so I've basically been able to do it solo for the most part because of Codex.

[00:33:48] So we can unpause the video but basically one of the first things I'll show you is that the reason my Codex is so helpful for me and go to market is that has access to all of the main systems that we use I mean Salesforce, Outreach, Slack, GONG, anything that I might need for high-level reporting or data pulling. Codex has full-blown access to it which makes a huge huge difference.

[00:34:02] So here this is an example of an automation that others mentioned that runs for me every Monday morning about like nine o'clock I think so that by the time I get out of my 9 a.m meetings I have a full-blown report waiting for me filled with like fresh data from Salesforce, GONG, Outreach telling me about the performance of our team at scale and then of course highlighting perhaps anything that may have been going wrong or trends that are looking weird or like off-putting and it even has the ability to go and dive into things that look troubling on its own or of course I can follow up in the Codex chat as well.

[00:34:41] This has been game-changing. Before this, like before this era, people like me

[00:34:46] Like people like me had to use a bunch of different people to support them, but rely on Salesforce reports. Salesforce reports, while cool I guess, are like cumbersome to build, but also if the underlying data change like maybe one of the reps is no longer on the program or something like that, you have to go and update the report. In this case, my codex is self-adaptive, I guess. It knows how to check the latest information and adapt what it's reporting on. So here, yeah, I'm just going to show charts. Basically, this is like one of my favorite features. It takes what I loved about Salesforce reporting in the past, bringing it into codex. Codex can not just only generate visuals, it can generate like charts and pie charts and whatever I need, really. I often find myself like screenshotting these charts and putting them in.

[00:35:28] Actually, if you can pause here real quick, this is just a screenshot of how it connects into my outreach instance using the outreach skill. It knows everything about our outreach instance. It knows how I love making sequences. It knows like our tone of voice. There's a lot of nuance that goes into making an outreach sequence, which is used for like email messaging. And my codex can go in about five minutes and make us a whole sequence from scratch that is effectively perfect. Like anytime there is an error, we just update the skill so that it never happens again. We can resume.

[00:36:05] But yes, at most organizations, someone usually works on this a couple hours a week. I barely have to. Lead assignments, pretty fun. This one, again, would require rev ops in the past. We get a lot of signals like job changes or stuff like that. Maybe teams are adopting codex or hiring for AI. Sometimes those are leads for our sales reps. So I use codex here in this instance to go find the latest signals, assign them out to our reps, like making sure it's all set up in Salesforce and then enrolling those leads and outreach sequences, which again would be pretty manual in the past.

[00:36:42] In this case, this is just me showcasing an automation that I might use. Others have mentioned automations. They are so important. Anything that I might do on a frequent basis is an automation and just tell codex, hey, make this an automation, just set it up for me. And it does it perfectly every single time. It's so awesome. And then of course, if I wanna click into it, I can go see the details and modify it should I need to. At most, though, I usually update the model to like the latest model, right? Like this one's on 5.5, I might switch it to like 5.6.

[00:37:15] This is another automation that has become very critical to our leadership team. We use this to do reporting, whereas in the past, other teams might use a newsletter or some kind of a PowerPoint deck, which might take a long time to generate. Codex has all the latest data from Salesforce. So it can go and make a website, which it does every week, and it updates the website every week on Friday with the latest information from our team, making it a lot more of a fun interactive experience. We can even track who's looking at the website and what parts they're interacting with the most to adapt how we're making the website. It's so freaking cool. It's like the frontier of technology.

[00:37:51] This is auditing. Again, these are like really manual processes. This is an audit of our outreach instance. Typically, you have to go, if you can pause it here, and you have to look through a bunch of different sequences. In this case, there's like maybe 30 to 40 to 50 outbound sequences, where it's like a rep reaching outbound to like a CTO or something like that. In this case, and you can unpause it now, Codex is looking through all the sequences, like balancing it against, of course, our standards internally for what is good and what's not good, and recommending like, hey, this one's not performing very well. This one's like repeating. This one, it could be optimized.

[00:38:30] In this case, I'll say, Codex, well, take a look at the best performing sequence. What is good about it? Go ahead and make a skill for this so that now in the future, whenever you make a sequence, it knows what are the characteristics and attributes of our best outbound messaging for sales. That comes down to skills. I rely on skills. Every single thing that you would need to know about like my business is a skill. As you can see, there's so many skills that are related to go-to-market. Most of these I made myself, like our BPO function, which I manage. That's a 2,400-word skill. It's like pages of information. It's so, so important, and anytime something new, I learned something new or something changes, I update the skill so Codex knows next time, hey, this is what we need.

[00:39:19] Anyways, that's all I got for you on my end. I'm going to pass it back to Allie.

[00:39:26] Hi, everyone. That was so great. Thank you for all the demos. It definitely gave me some ideas about what I can do in my work in global affairs. We are going to, we actually, I'm sorry for our community. We are a little bit over on time, so we're going to just do one question that I'm going to pose to everyone.

[00:39:44] and I'm gonna pose to everyone here and then so you can all answer it. [00:39:49] And I'd like to start with Drew. Drew, what is one idea you hope everyone here leaves with today? [00:39:56] Yeah, I hope that take away that everybody leaves with from this is just that these things are moving pretty fast and that the technology is there for agentic AI to really change workflows outside of open AI as well. The limiting factor there is more organizational buy-in and efforts internally rather than the technology itself. And I've seen that open AI that once the ball gets rolling on those things, things can move pretty fast.

[00:40:29] Yes, I do agree with that. Melanie, how about you? [00:40:31] Yeah, I think the biggest thing is that if you're not sure what to do or how to start, just ask. Like, ask the most kind of like moonshot question you can and then figure out where the failure points are and solve those. Because usually it can do a lot more than you necessarily think from the outset. Yes, I like that idea. Or what's the most annoying task of your job? Solve it, automate it for me.

[00:40:58] Blaine, how about you? What's one idea you hope everyone leaves with? [00:41:03] I hate to reiterate what Melanie said, but I think that was the big breakthrough for me of just saying like, hey, how do I do this thing? I need this solved for me. I need a, how do I make a video of this prototype? And Codex can help you figure out how to use Codex. And that's been a really huge unlock just changing my mindset that way.

[00:41:23] Yeah, Ronnie, how about you? What's your big thought for everyone? [00:41:28] I think the data shows that AI is changing the way we work. I think the second part of this is gonna be organizations are gonna change the way they get work done. And that's the process you're seeing all across the enterprise. Now that we have all these new ways to work, to be more efficient, to do more things at the same time, to build skills that are scalable, organizations are gonna change the way they get things done and that's a process gonna lead to a lot of changes in the economy.

[00:41:51] Love that. Ken, what's your closing thought for everyone? [00:41:55] I think Codex performs well at a base level and then it performs even better the more context you give it about yourself, your work, what good looks like, what bad looks like. The more information it has, it's like a colleague, right? If a colleague is onboarded effectively, it'll do a really good job. But if you tell a colleague, get in there, I'm not gonna tell you what good looks like, it may not do what you think is gonna be the best work and that's what's worked best for me.

[00:42:22] I love that. Well, thank you all so much for carving out some time in your day to join us and thank you to our community as well for tuning in and for the great questions. I'm sorry we couldn't get to all of them. But we hope this sparked some great ideas. Keep an eye out for more forums even in July, we're about to announce two events, one on AI and scientific advances and another on how AI is helping doctors diagnose rare pediatric diseases.

[00:42:48] So we're really excited for all this. So stay tuned to the forum newsletter as well as to the OpenAI Global Affairs LinkedIn page for more. And thank you all again and hope to see you in the OpenAI forum sometime soon. [00:43:04] Thank you.