I'm Natalie Cone, your OpenAI Forum community architect. I like to start all of our talks by reminding us of OpenAI's mission. OpenAI's mission is to ensure that artificial general intelligence, AGI, by which we mean highly autonomous systems that outperform humans and most economically valuable work benefits all of humanity. It is always truly an honor to welcome you to our community sessions. And it's a special, special honor to welcome all of you who are joining us from different time zones and even from your vacations. We're grateful for your participation.

Today's discussion will focus on how Wharton is becoming an AI-native institution and how it all started with one forward-thinking professor and deputy vice dean of the executive MBA program, a couple of MBA students or super TAs, as Richard would like to refer to them, and an IT project lead, and how this small group of people have catalyzed AI adoption in their institution.

But before we get started, I'd love to hear from you. Please drop in the chat what you're most excited to learn about today. Have any of you tried to launch an AI project or initiative in your university classroom or a club? Feel free to share your thoughts in the chat. And throughout our discussions and presentations tonight, please also start to leave your questions in the Q&A tab.

So tonight we'll hear from several innovators. We'll start with my colleague, Sia, move on to a fireside chat with Dr. Waterman, then bring his collaborators onto the stage. After the presentations, we'll open up the floor to member Q&A.

But first, let me share a little bit about our speakers with you.

Dr. Richard Paul Waterman is a practice professor of statistics and data science at the Wharton School of University of Pennsylvania, where he focuses on the application of data science techniques and AI technologies in an educational setting. With a PhD in statistics, Dr. Waterman has dedicated over three decades to pioneering integrative methods that enhance teaching, learning, decision-making, and higher education. He teaches in the undergraduate, MBA, and executive MBA programs. He works on applications and projects that aim to transform educational strategies through predictive analytics, machine learning, and AI. Dr. Waterman has authored numerous papers and books in statistics and data science areas. As the deputy vice dean in the executive MBA program, he seeks to identify and implement AI-driven solutions that drive engagement, enhance student learning outcomes, and streamline administrative processes. Dr. Waterman's collaborators are MBA students in his courses and an IT project lead from Wharton. They'll be joining us later in the talk, but I'd like to introduce them to you now.

Nupur Jain is a product manager and executive MBA candidate at the Wharton School. With a background in mathematics, economics, and computer science from the University of San Francisco and the University of Illinois, she's held strategic roles at companies like Gap, Google, where she led initiatives in business operations, security, product development. For her data-driven approach, Nupur has driven key projects that improve operational efficiency, mitigated risks, and generate significant revenue growth, all while fostering innovation at the intersection of technology and business.

Ceren Okar is currently pursuing an MBA at Wharton. She holds a graduate degree in mathematical statistics and an undergraduate degree in business economics. As a financial controller, she's gained valuable experiences across industries such as housing, education, health care, bringing a unique blend of analytical and financial expertise to her roles.

No wonder Dr. Waterman calls them super TAs. Those are some pretty super learners.

We'll also be inviting Brandon Lafving, who grew up in Dallas, Texas, studied comparative literature, mostly poetry, at Princeton, and somehow fell in love with technology along the way. He can't explain it, but he blames it on sci-fi and the very real exciting progress his generation has experienced, from IBM's Deep Blue to AlphaGo, and now an exploding assortment of ML solutions. He works for Wharton School at the University of Pennsylvania as an IT project leader, identifying, planning, and operationalizing valuable change across technology, people, resources, and processes. Over the past year, his projects have increasingly focused on large-language model-based chatbots and applications. He's led product development for two course chatbot prototypes, experimenting with a variety of approaches and platforms. He is passionate about automation, machine learning solutions, and new challenges.

But first, we're going to hear from my brilliant colleague, Sia Raj Pruhit. She's an education leader at OpenAI, where she helps manage the growth of chat GPT EDU across higher education. She is spearheading transformative efforts to help universities harness AI's potential for more equitable, impactful learning experiences. And honestly, Sia is the reason why we are all here today.

So please welcome my colleague, Sia.

Hi, Sia. Thanks, Natalie. And hi, everyone. I'm so grateful you all took time this late in the evening on the East Coast to join. I'm Sia, and I work on the education team at OpenAI. I'm going to basically provide a little bit of an introduction in what OpenAI is doing in education, how we work with universities like Wharton, and kind of what our current thinking is around the future of AI-native institutions. And then I'll pass it back to Natalie, who will introduce the panel and move forward.

So to give you all some context, a big part of my role is to work with universities to help them develop an AI vision. So I work with presidents, provosts, university leaders to start thinking about what their AI vision for their university is. I work with professors who build custom GPTs and AI solutions at the university. And then I work with students who actually drive a lot of that engagement of AI on campus. And while doing all of this, I'm realizing that there are three stages to workforce transformation, especially in education. And basically, how we are going to move the sector forward is by going through these stages.

The very first stage is at the individual level. This is like specific professors who start building solutions to help solve their own problems. A professor told me that he has built a custom GPT to help write letters of recommendation. He's like, I get so many requests for writing letters of recommendation that I've learned how to scale it with custom GPTs. So the custom GPT is trained on his previous letters of recommendation. And he adds in a couple of bullet points about the different student that he's referring. And it creates a new letter of recommendation in his voice. So it saves him so many hours per week. But the very first level, basically, is individuals starting to use chat GPT to help solve their own problems. That then moves up to the team and department level.

I recently learned how cumbersome a task it is to plan out which room, which course should be in on a campus, because the campuses are so big, there's so many courses and so many rooms. That it takes a team of like 40 people to help decide that before a semester starts. But that's something that chat GPT can aid with much more easily. So the next stage of transformation is when departments start working together to solve these kind of challenges together to make themselves more effective. And then the final stage of transformation is at an organization level. That's when universities like Wharton have decided to add so many departments.

different AI touch points across campus to make the experience much better for faculty and students. In the future, what we envision is that we'll have AI touch points from everything from like onboarding and orientation GPTs up through GPTs and classes that help students understand and interact with the knowledge of the classroom more effectively. GPTs and career services that help them like prepare for the jobs that they want to, practice interview with like chat GPT, and then finally across the clubs that they have as well. And hopefully in the future, students will be able to seamlessly move between these GPTs, creating a really easy experience for them to engage with all of the intelligence of a campus in a conversational way.

So that's our current thinking about what the future of AI Native Universities is like. We're really excited because we hear every day from professors the innovative ways that they're using this in the classroom. Uploading their case studies and content and just letting students talk to it about different problems that they're having. We'll see a lot more of these examples soon. But I'm really excited because I think we're moving as an industry from the individual level of use to the department level of use. And then universities like Wharton are leading that organizational level which I hope that other universities will follow. So I'm gonna pass it back to Natalie who will introduce the Wharton team. And if y'all have any questions, I'll be around later today.

Sia, thank you so much for that. And thank you so much for bringing this awesome group of people into our fold. So now we would like to invite Dr. Waterman to the stage and discuss his initiative.

Hi, Dr. Waterman, welcome to the Open AI Forum.

Thank you very much, indeed, Natalie. It's so good to have you here today. And I just have to mention, it's actually been a real pleasure getting to know you, your students, Brandon. You have clearly really brought together a really awesome and kind group of people. And that says a lot about yourself as well. So thank you so much for inviting everybody here today.

Too kind. So Dr. Waterman, let's start with the origin story. I loved your origin story. What made you embark upon the journey with AI adoption in the classroom?

Well, it was a day I was out in our San Francisco campus, and I was there in my administrative role. So just basically walking the corridor saying hello to people. And I happened to look inside one of the classrooms where one of my colleagues is in a different department, was teaching his class. And he was talking about something like supply chain shocks, very different from the sort of thing that I might talk about. But I looked, and I could see what was projected up on the board. And I thought, my goodness, he's using exactly the same ideas that I'm using in my class. And then I thought, gosh, I mean, I wish I'd known that earlier on, because then when I put my class together, I'd be able to refer to him, I'd be able to think about integrating the materials that I had. And so that was the, you know, the observation that I made.

And then it suddenly occurred to me that at Wharton, every single class is recorded, not always shared with the students, but it's always recorded, and also has a transcript. And that just popped into my head at that point that well, maybe there'd be some way of looking through these recordings. But you have to think of how many there would be of those. So typical class is about 18 hours, we might have 120 classes going on over the year, that would give me about 2160 hours to watch if I was doing it as a human, you know, more than a year's worth of labor. And by the time I've done that, it would all be changed anyway. And it's simply not feasible for an individual to take that approach. But obviously, we have a lot of talks around, in particular, LLMs these days, that will be able to ingest content. And essentially, in this particular instance, you know, workers in really enhanced search.

So I thought, well, wouldn't it be nice if I could just type in a query that said, show me all the classes in the core curriculum that use such and such an idea. And hopefully, back comes some response. So that was the initial idea. And we can talk more about how we started to implement that. But as soon as we had a prototype up and running, it became clear that it would be extremely useful for students as well, it was almost impossible not to want to be a student and start querying it. Things like summarize a particular class, tell me where the professor talked about Bernie Madoff, what is likely to be on the exam, all these sorts of questions that I might, in some sense of commented on in class are, you know, of interest to students. And so that was the, you know, the basic idea of, well, let's try and create some of these chatbots that I call statbots, because that is obviously my discipline. And, you know, that was the motivation. And luckily, at Wharton, you know, there's a lot of interest in AI and a lot of support for doing that. And I was introduced to Brandon and some of the IT team and just started to go from there.

That is awesome, Richard. And can you tell us a little bit about how you inspired the other faculty members to buy into this idea and contribute?

The I mean, we have two projects. So one of them is called Wemba Insights, which is for the faculty, a faculty tool, so faculty can discover what else is being taught and hopefully integrate with other courses. So there's that one. And the other one is the statbot, which is student facing. But in terms of the faculty one, what it involved me doing, and this is sometimes advantageous to be an administrator, because I know all the faculty, and in some way, I decide what they're going to teach. I wrote to them. And in our core faculty, we had about 36 of them. And I said, here's what we're interested in doing. Would you like to opt in? And 18, roughly 18 out of 36 said yes. So I mean, it's not everyone, but we have a lot of people actually who don't need a lot of convincing. And they would say things to me like, why have we taken so long to do this? You know, so and so there are plenty, I have plenty of colleagues who it's not arm twisting to get involved. It's, you know, because they're interested and see value.

And do you think Richard that you would have been as successful? We have a lot of faculty online tonight, as well as administrators. Do you think that a faculty member who's not an administrator could also drive adoption in this way?

Well, yeah, definitely. Definitely. There's no doubt about it. Because I mean, you don't have to be an administrator. It just gives me some perhaps access that I might not have had otherwise. But I mean, it's a little edge, a little authority. It's not a prerequisite, for sure. I mean, I think there are, you know, that there's support, and the institution has, you know, bought into it, in the sense, somewhere money has to be spent at some level, behind these initiatives, they're certainly not free. And so at certain levels of the university, people have had to buy into this. And at Wharton, you know, that isn't a problem.

And so would you say that the 18 faculty members that did buy in, are they actively using it?

No, I mean, the way that the two projects went is that the student one really took off very, very quickly. So the other one is sort of catching up, I would say. But the student one is the one that's out there in the wild and getting lots of use as we speak. But other faculty are picking up on that now as well.

Yeah, that's actually a really interesting data point. Like, maybe the folks

that are here, like if you're really trying to drive adoption, maybe speak to the students, like really lean into the students.

Yeah, I mean, I think the thing about the students is that they will eventually almost be demanding these sorts of tools. I mean, the pressure that the students can bring to bear as a group, I think, you know, they might underestimate how much strength they actually have, but professors at some level tend to be quite responsive to student needs or requests, and if you're the only professor who's not doing something, you can find an awful lot of pressure to start doing it, and so I anticipate that over time it will be like, you know, a force against a dam in some sense. The dam will break, and there will be an awful lot of adoption very quickly.

Oh my gosh. Dr. Waterman, I think you're going to be inspiring a lot of graduate students in the audience tonight. They're going to be very excited.

I don't want them to hassle their professors. Please don't take that as the take-home.

Well, will you tell us a little more, Dr. Waterman, about the student-facing chatbot? Like, why was there such a demand for it? What are you guys specifically working on right now? What are maybe some of the challenges of scaling it?

Right, I mean, you know, students always, I mean, if they're good students, are trying to keep up with the material. They're studying for exams. They're thinking about quizzes and homeworks, and, you know, that's on their mind, and they're always looking for support to help in that endeavor, and in particular, the group of students we have here, the executive MBA students, they in fact have full-time jobs as well as doing their MBAs, and so my sense is, and they can corroborate me or not later on, is that anything that makes them more efficient in their studying is a valuable tool. I mean, and that's the bottom line, and that's what this chatbot is able to do for them because it ingests, I mean, I told you everything we have is recorded, and so it will ingest the transcripts from those recordings. It will ingest my notes. We're having it ingest the exams, the slides, etc., etc., and then, you know, with that ingested, you know, queries can happen, and all sorts of interesting questions can start to get answered, and they're answered very, very quickly, so from a student point of view, they might write me as their professor, you know, I don't understand this professor. Can you help me? And it might be a couple days, if we're lucky, before I get back to them because, as many folks, my email is rather thick, but if they can type in a query to the chatbot, it might take five or ten seconds, and that, I think, in its own right is just a huge advantage, certainly for the sort of learners that we have at that level.

And so, Dr. Waterman, it sounds like you didn't have to engage in much training for the students to adopt the chatbot. Is that true?

Well, that is where Nupur and Saren come in because I am so lucky to have such people around, and so, I mean, they can talk to that in a minute, but they are really driving the student engagement, and I'm, I would say, very, very lucky that I have them on board, you know, and they're obviously interested, committed, and motivated, and that would be something else I'd say to any of the other faculty that are listening out there, that having, I would just call them evangelists, you know, they're evangelists for what we're doing, and having people who are willing to step up, talk to their classmates, I mean, you know, peers are probably the, you know, the strongest incentive to try something new, and having them as peers, understanding what they're going through, I think, has turned out to be invaluable.

That is very, that's just lovely, and also, and one of your students is an executive MBA, as you mentioned, so this person has a...

Both of them are, to be clear here.

Oh my gosh, wow. They're both executive MBA students. So they're both working full-time, wow, and they've taken on all these initial, all these additional responsibilities. I can't wait to talk to them, but, um, okay, so what about the faculty, Dr. Waterman? You said that's a little bit slower to catch on, but is there...

I would say our two projects are a little bit different, right? There's the faculty-facing one, which I think, you know, we are not giving up on, we're continuing, but the student-facing one is the one that has really taken off, and it has an immediate, obvious benefit, as soon as it was out there. So, you know, it rolled out in September, and Brandon was sharing a graph with us today, and it had that hockey stick type look, in terms of adoption across the students. And so, you know, that's sort of, I don't know, I don't want to say easier, but there's just an immediate, immediate demand for it. And so what I was saying in terms of faculty is that I understand now that, so my class is the first one that's had it, but in the next quarter, meaning about now, other faculty are going to start bringing that in, that same infrastructure and architecture into their classes as well. So I think that's something that Brandon will be able to speak to, is that the ability to scale it, and I think that it has been built with an eye to scaling it, and that's the beginning of that process, it's starting to happen.

That's very exciting, Richard, so exciting. And I think also, I really love, you know, you suggested this other tip and trick, and that is not to give up. So adoption was a little bit slower for one of the initiatives, but you're not giving up, and teachers are slowly, they're adopting it a little more slowly, they will start to see the advantages over time.

Right, I mean, everybody's resources are somewhat finite, and you have to decide where you're going to prioritize, and it just sort of naturally evolved that, you know, a student-facing product is just, you know, you've got hundreds of students who are going to be interested in using it straightaway. So, and I think, you know, we learn an awful lot through that process, and what we're learning through that process will now get applied to the faculty-facing Wembley Insights tool. Though, I say we haven't given up on the tool, we're working on it, but in terms of a race, it's in second place, perhaps, and the student-facing one is in the first place. And it's not a race, so they're both successful.

And would you say, Dr. Waterman, that in general, throughout the history of your career, are you more inclined to be one of the first people to adopt a new technology, or is that sort of ingrained in your DNA? Is that your approach?

A little bit. I was doing video things back in the 1990s, you know, CDs and things like that that don't exist anymore. There was a math CD at one point in time, so I was always game to try something. I mean, I don't really see much of a downside, and you know, if I'm being absolutely truthful, I get bored kind of quickly, and so I love trying new things.

I love to hear that, Dr. Waterman. So, before we bring on your amazing students, could you share what is your vision for the future? Once these chatbots are scaled, maybe we discuss the one for that student-facing first, since that's the one that's really caught fire. What does that look like in a year? Well, a year or two years. I mean, maybe I can go out three or four years, because I can imagine at some point where it's just a standard part of every class, you know, and quite importantly, the faculty doesn't have to do very much, because that turns out to be a bit of a deal breaker as well. So, I'm sure we'll get very, very good at automating the process, you know, get some processes in place to help the professor get this going. But, you know, I can imagine a day where it was just a totally standard part of every class, just another...just like we have LMS, the Learning Management System, just as every...

class has that now. So every class will have one of these chatbots. I mean, I find it hard to believe that won't be the case. They might not all look the same. They might have different objectives.

If you're teaching a class that's very case-based, you know, that's a different kind of learning experience than my class, which is a stack class and lots of technical facts and skills. But I anticipate that they would be ubiquitous. I can't see how they will not be ubiquitous.

We will be here for you to help you see it through to the end, and we're so grateful that you took the time to share your experience because I think a lot of other faculty members and administrators are excited, optimistic, but they don't know where to start. So thank you so much for sharing.

If I could just add one final thing, Natalie.

Of course. Our other initiative, the Wembley Insights one, which is going to be for the faculty. I think if that works, that actually might really change profoundly how we do education, to be honest, because it's always been a holy grail to integrate parts of the curriculum.

And I know we have tried to do that at Wharton before, and it's just not, or didn't used to be practical given the amount of time and energy it would take to do so. But with the tools that we have now, I mean, we are going to be able to kind of almost redesign this curriculum around a much more integrated approach, and we're going to be able to just kind of learn about everything else that is going on and put it together. And I think if that happens, that actually will change fundamentally how we teach.

So that kind of, it's like me a lot that that possibility is out there, but I think we might be lucky enough to be the ones at the time where we finally have the tools that can make that happen.

That is certainly huge. And if there was one blocker that if you could just magically get it out of your way so that that project could really take flight, what do you think, like, what is the blocker?

Maybe human nature in that not everybody wants to share themselves with everybody else. I mean, down at the level of a professor might not want anyone else to see their jokes. You know, they're special. They don't want someone to steal their jokes.

I mean, that's a little bit silly, but there is an element of that to us as humans. This is what makes me special. And I don't want to share it with every colleague because they'll be able to take these, they'll be able to see these pieces of me. And so I think that is something, if we could remove that, the adoption would go a lot faster.

Okay, that's super interesting. I want to share something with you now that that reminds me of. About six months ago, we hosted Karen Kimbrough, the chief economic officer from LinkedIn. And she shared a graph that kind of demonstrated where they thought like what careers would be augmented by AI, what careers might be supplanted, what careers would not be touched at all. And what careers like what skill sets and careers would be deemed more valuable in an age of very prolific, large language models, generative AI.

And it was those human-focused careers, like community building, or being a professor, like the face-to-face engagements, those are going to become profoundly valuable.

So I don't actually know if a chatbot, and of course, it doesn't make, whether or not it's true, it doesn't change how we feel, we could still be anxious about things that, you know, would not actually ever come to be in the future. But I actually don't think that professors jokes like nobody can tell them the way the professor told them. You know, just just reminded me of that.

I don't think any, I don't think a chatbot would ever be able to replace, or any new design in instruction would ever be able to replace the value of that face-to-face time.

I don't think that was the concern that we would replace them, but not everybody wants to open themselves up to everyone in the world. So this True. True.

Yeah. Oops. We just lost audio from Dr. Waterman. But while he's figuring that out, and also, Dr. Waterman, just remember the settings icon in the bottom left center of your screen, potentially, that is it.

But while we wait for Dr. Waterman to get his audio working again, let's bring on his amazing executive MBA students, Nipur John and Chetan Alkar.

Ladies, so good to see you. Hi.

Thank you for being here.

And it sounds like you both are so incredibly busy, but you're here. So thank you. Thanks for inviting us.

Yes. So my first question is the obvious question. You guys are both working professionals in leadership roles. You're taking Dr. Waterman's classes, and you're in this executive MBA. What in the world inspires you to take on this additional role? Because Dr. Waterman calls you his super TAs. You're his teaching assistants in his mind. And you guys are really driving student engagement.

So maybe we'll start with Chetan, and then I'd love to hear from Nipur. But what inspired you to jump in on this project and invest more of your precious time?

Sure. I had the initial conversation with Professor Waterman, because I already have a master's in mathematical statistics. So I was in the process of waiving the class. And he mentioned about this opportunity, and I'm very much interested in education and the AI in education. So I kind of jumped to the idea, and I said that since I'm waiving your class, I will have extra time. So I'm more than happy to help you.

So it is a very interesting project, and I'm very happy with the outcome. And I'll let Nipur to speak to her part.

Thank you, Jaren. Yeah, I was not fortunate as Jaren to waive out the class. So I am definitely taking the class, but I'm very much involved in the AI field, Natalie.

So for me, the astute observation was half the class was already using Tattoo PTA as an aid for classes or learning. And so what Professor Waterman did was just raise the bar and made it more custom. So it was very much focused on learning for statistics. And so I just, for me, it was an opportunity of once showcasing that, hey, I can help these students for those who are on the edge.

But also, it's a personal interest and investment. Like, what does it take to actually build this for a pretty big class like ours?

Also, people with various degrees of understanding of LMs in general. Like, some people are very scared. Some people just don't use it. And some people are just heavy, heavy users. But what would be helpful for all of them? And that intrigued me a lot. So I reached out to Professor Waterman, just like, you have to get me on the team right away.

Awesome. Thank you so much, ladies. And Nipur, let's actually, I'd like to pull that thread a little bit. So in an executive MBA program, there are a lot of different types of professionals, people from different backgrounds, different educations, different professions.

And we often do see a divide between early adopters, those who are skeptical, those who are more cautious. Can you share any trends that you've seen with the students? Like, which was there a certain domain about?

expertise that was more open to adopting versus another or any other trends? So I will say because we are dividing the West cohort and the East cohort, the West is a little bit more tech based where it's just playing into our, you know, our field of expertise here. And so I saw a lot more adoption, natural adoption from the West cohort. So people were just using this even before we introduced the chatbot as well. East Coast is a little bit more finance heavy. While they are using it, I think there were some initial skepticism like, is this helpful? Is this an age? What should I be doing with it? I don't use this for my work because that's not generally where I would typically use it for. And I think we just needed a little bit more handholding with them. But I definitely see the West cohort winning this round, particularly in adoption. Shaden, anything to add to that?

Yes, I mean, to your point, we have a very diverse group of students in the classroom and they are highly educated, highly skilled, accomplished professionals already in their own chosen field. I mean, we have doctors and lawyers and Ph.D. level engineers and they really take pride in their work. And when it comes to academics, they often choose the traditional methods of learning because they know that it worked in the past and they don't want to really deviate from that. So these are what we call like skepticals. But on the other hand, they're also intellectually very curious and they want to explore new technologies regardless of their chosen field. So those will be called the early adopters. So what happened kind of organically for our use case that the early adopters use a system for their homework and quizzes and then they talked about it and how it was helpful for their own studies. And then the skepticals kind of jumped on the board and they said, like, OK, we want to try as well. So right around that time, Nupur and I, we scheduled an office hour where students can join, ask questions. And then we did a video and then we made it available for all students. And I think it helped a lot with adopting the new users. And today, Brandon shares some numbers with us, like how many people are actually using it. It's about like 70 percent of the entire, like both East Coast and West Coast are currently using the chatbot for the class, which is great, actually, because it's been just like a month and a half that we started using it. So we have two camps if we were crafting a playbook and they're the early adopters and the skepticals. And there's actually an approach for working for both of them. And when we download the transcripts of this talk and we pull out some of the key moments, we'll be able to identify those and share them with you. You'll have a little bit of a playbook. That's always fun. Just a note, Dr. Waterman, we dropped a note for you in the chat, like potentially some advice that might help. I'll continue chatting with the ladies, but we want to try and help you get your audio set back up. Has it come back? Oh, there you are. You're back. OK, great. Awesome. All right.

OK, so for the student-facing chatbot, folks, how do you effectively communicate or like set the expectations? Because it is possible that the chatbot is not going to be 100 percent accurate. So are you setting the expectation that this is a tool and not the Holy Grail? What does it look like to set those expectations? And what are the expectations that you're setting with the students? Maybe we'll start with Ceydin this time, just to switch back.

And thank you for asking this question, because the expectations should be set during the implementation or even before the implementation by the professor. Because again, in our classroom, we are very busy professionals and we want to be very efficient and effective with our time. So the questions that students are typically asking is the first thing is, is this mandatory for the class? Like how is it going to impact my grades or like what am I getting out of it? Like what's the value I'm getting from using this platform? And also the actually the question that they ask the most is that how is that different from other platforms available out there, especially from a regular chat GPT? So those are like the main ones that are out there. And the third one that it came out, is it anonymous? Because after the usage, they're very reluctant to use it if they know that their activity will be seen by the faculty. So they're very cautious about that because they don't want their mistakes to be seen by the professor, by the faculty or whatnot.

I think the main expectation for us during the implementation was to emphasize the feedback mechanism, because the user feedback is really crucial for us to have fine tuning the system. And related to that, I think the faculty needs to set the expectations from the beginning on what the system can do and cannot do, especially if we are in a quant-heavy class. And then the emphasis on its limitations is very important and around like computations or graphing, because we have another software that we use for the statistical software and then where we actually analyze the data. And we always emphasize that, do not try to do that in the chatbot, please use the other statistical software. And then the results, then you can come back and then kind of get the interpretations that was discussed in the class. And then we always make sure to emphasize that this is not going to be 100 percent correct all the time. There will be errors. And as you know, like human, when they make errors, that's OK. But if a computer makes an error, it's oh my God, like it's the end of the world. They give up. It's frustrating. And we get that. But if you set the expectation beforehand about its limitations, I think it's more forgiving. So that's why we always refer to as a lecture recall, meaning that everything that is being discussed in the class, we can recall that information rather than a TA replacement. Like if you still have questions around the quantitative area, about the calculations, we always refer to them, like please use the TA or go to the professor to get actually the calculations. But not like if there's a simple maybe like what is going to be asked during the midterm, something that was discussed in the classroom. You can always refer back to that during the chatbot or like the interpretation of the graphs or whatnot. So we set the expectations constantly though, way from the beginning, Oprah and I have been communicating that to the students.

Of course, Richard, I think one of the fundamental words here is trust. I mean, that's what students need to eventually be able to trust the technology to some extent. And one of the key features of what Brandon has helped build for us is that it will because it has access to all the transcripts of the videos, it can recall the part of the class and also provide a direct link to that part of the class so the student can watch the video. So they have some ability to corroborate what is being returned by the chatbot. So these live links directly to the piece of the class come back. And I think that's one of the really distinguishing and important features, because if a student isn't sure, did he really say that? Is that what he meant? They can actually click on the link and lo and behold, I pop up that precise part of the video while I'm talking about it. So I think that's, in my mind, something that we have to, you know, deal with. Definitely.

It's all, in the end, going to be about trust from the students.

Okay, awesome. Well, we can't wait to chat with Brandon about that.

So in an ideal world, you also have a technical stakeholder also supporting the project in the background in order to build trust, make sure it's working for the students, it's reliable.

And then lecture recall. So we're naming it something that sets the expectation purely by the way we're naming it and then setting the expectation early and early and repeating. And I think the operator in me really hears that part, folks, because anytime you're establishing any sorts of practice, practices, routines, and rituals, it's not a one and done deal. It's over and over and over again. It's not creating a document of processes, dropping it in the drive, and then your job is over. It's actually reintroducing it over and over and over. So all of that really speaks to me.

Nupur, is there anything that you wanted to add on that note?

Yeah. I know we spoke about the restrictions and I was observing the chat as well and the questions that were coming in. And I also wanted to highlight how exciting this was for our students as well.

While we knew that this was more of an aid with our classes, the possibilities were just amazing because we started going into just making sure people understand how to use prompts properly and what kind of things.

But, you know, we had students who were just like, please explain this to me like, you know, like a five-year-old child. I haven't done statistics in a very, very long time.

I've had students and like fellow peers who have just been like, I don't understand a particular concept. I know Professor Waterman taught me this part. They'll go to the lecture and then they'll say, can you create questions for me to explain and deep dive into this topic? And then they would get questions from the chat bot.

Like we I've been working as you're and I've been working with students just to figure out different prompts like you can role play. You can be like, imagine if I am a TA for this class and or I'm a fellow peer and I want to talk about this particular concept. Can we do that?

So I think the possibilities are kind of endless in that sense. But the more you think of it as it as an aid at your at your almost like a doorstep, I think that that's the thinking and the shift that happened with our students where their usage was more from early adopters to like, oh, God, like I can do all of these things. I had no idea. So like I need to start using this ASAP.

Ladies, that was so wonderful and we could honestly chat all day. I would really love to stay connected with you. Let's have virtual coffee.

I want to give Brandon the opportunity to contribute a bit, but I would also there are a lot of questions, so we're not going to be able to get to all the questions live. I'm going to share them all with you later. I know you're so busy, but if you have time and you want to respond to the ones that seem applicable to your experience, we'll also give the audience more of an opportunity to hear from you in that way. Thank you so much for joining us, ladies.

So now I'd like to bring on Brandon Leifeng and Brandon, I'm so sorry. I think I am not pronouncing your last name correctly. As we discussed, can you please remind me how to say your last name correctly?

No, that was perfect. It was the first when you introduced me in the beginning, it was incorrect and this was right.

My brain. Brandon, welcome. Thank you so much for joining us.

Glad to be here.

The first thing I want to ask is, well, let's touch let's go back to this notion of building trust with the students. So from a technical perspective, what are you contributing so that the student body can really trust this tool? What does that work into?

Well, so I guess really there are a few different parts to that. So first, we handle authorization. So we on a on a systematic level, we know what who the student is and that's what we use to look up, make sure they're going into the right database and they have access appropriate to their actual section. So that's very specified. And, you know, sometimes the bot will greet the student by name and because it has that authorization. And I think that that helps. Right. They know that this is a personalized experience.

And then in addition to that, we touched on this before. You know, we really put a lot of effort into making sure that we that students could look under the hood if they if they want. Right. Like so all of the things that, you know, the bot will under almost all circumstances just send a query through our action to the to a vector database.

And that data comes back with the full quoted chunk. That's a precise quote from the caption. And and it also has a timestamped link to the video like Richard was talking about. So they so they can they can ask. They can say, oh, I don't really trust this summary or exactly how the bot is talking to me. Like, just show me the quote, you know, or they can click on the video. They can they can do all of these things to reestablish that trust to really corroborate what's going on.

And so that's that's very deeply a part of what we were trying to do. When we were bringing this sort of specialized knowledge into the chat bot to make sure that I think and then in addition to that, it's also a little bit the system prompt and making sure that the bot has rigid instructions to rely on the on the action in our in our database and in the transcripts instead of its training data, because then, yeah, and I can I can go into that, too, and some of those differences. But I think that answers your question.

Oh, wait, I think you're muted, Natalie.

One thing I wanted to just draw our attention to is that your background initially was in poetry, and so I think that says a lot because now you're very technically inclined and you're talking about really fine tuning and training the bot. You're talking about making the mechanism that might be very implicit to somebody who's technical, explicit to people who are not technical.

And your background didn't it's not rooted in a technical domain. So I think that should be encouraging for everybody here, including myself, that it took me a while to adopt AI and integrate into all my workflows because we all have this sense that this technology is for somebody super technical. But we can all learn how to do it.

I just wanted to bring that up because I think that's very fascinating, Brandon.

Let's take a few steps back and I'm curious, how did how did how did Richard approach you and ask for your support? Like, how did the collaboration kick off?

Oh, wow. I'm trying to remember. We started to we started to meet this was early, early summer and talk about Wenda Insights. It wasn't you know, we didn't have names for these things.

We started talking about these ideas and, you know, we just started a meeting cadence. I was I've been super interested in the concept of AGI since I was a kid and been very interested in open AI and all of these projects. So like meeting someone who is also who also shared that interest was was very refreshing and.

So we just started to work together and really figure out how we were going to architect this stuff. So that was it. It was good. And I know these things can be very iterative and maybe even slow-moving, but can you describe any sort of breakthrough moment during the development when you realized that the tool was truly going to exceed expectations and you guys were building something very exciting and useful?

Oh, absolutely. So technically, we had been, you know, our lead developer, Sean Zamacek, had come back from vacation. And all of the best thoughts happen on vacation. And he was like, hey, I want to try sort of a synchronous chunking approach where instead of just up to that point, we had been cutting off each chunk for the vector database every like 10 or 15 lines. He was like, why don't we try doing a similarity score, cosine similarity score for each line so we get more organic chunks, right? Like where there's a natural break in the conversation. That'll be the end of the chunk, right? So we'll get a more topical, more consistent, intuitive database. I thought that was a great idea. And we did it. And then once we did it, I remember that conversation with Richard because I had started to test it. I was excited. I was like, these results are so much better. And that was the day Richard was like, I might be out of a job. This is great, you know? And do you remember that? It was like, it was really a big, it was a big difference. And, you know, all of the other technical stuff was there, but it just was that next level where it felt like the answers were much better grounded in topical understanding and just raised the bar.

That's very exciting, Brandon. And congratulations on that project.

Brandon, there's so much I would love to talk about. We're running out of time. We want to leave a little bit of time for the audience. I want to remind you and Dr. Waterman and Nupur and Jeren that in the wake of this event, the chat history from this event will live in your messages. So if you want to engage with the faculty members, the OpenAI team that are here, anybody that's present, you guys can continue. So, Brandon, I know there's a lot. There's still a lot left to this project that you have to communicate, and I would love to invite you back, and you can share that with the audience async. But before we move on to audience questions, I want to give you the opportunity to pay homage to your collaborators. I know it was important for you to share a little bit about the other team members that helped you guys stand this up. So, please.

Thanks. I really appreciate that. Yeah. I mean, it's really difficult at this stage of any technology to really prioritize this work because the ecosystem doesn't exist. We had no idea that students were going to like this. Everyone has different opinions. And so our team, like, just the fact that we were able to prioritize this and really put a lot of good work into it is just incredible. I already mentioned Sean Zomacek, who is, you know, the heart of development for this project. James Mason also made a lot of contributions. And, you know, and we had also our, like, leadership at Wharton Computing was also, like, we couldn't have done it without that. We really found leadership creating a bubble in which we could function and where we could create this. And I think that I'm so lucky to be at an organization like that where we can find and create that space.

That's really rad, Brandon. And I echo that. It really sounds like you guys are part of a very supportive organization and everybody dreams to be on a team like you guys are on. I've just dropped the member referral application in the chat. It's for you, Brandon. I know you also have it in your email, but you can grab it just so it's top of mind and you can bookmark it in your browser. But you can invite all those people that you mentioned from your project team to the forum and we can host them for a roundtable. We can continue the conversation. It doesn't have to end tonight, so please go ahead and use that. And we're going to move into audience Q&A, but before we take Brandon off the screen, we actually have a question that relates to him. So Marissa Sadler Holder, Hendrick, if you could spotlight Marissa, if she's still here, has a really great question. It's related to improved student outcomes and how you guys are tracking that. I can also ask her question. So, Brandon, she asks, have you seen improved student outcomes since integrating? Are you able to capture that data?

There's Marissa. Hi, Marissa.

Hi. I'm sorry. I don't know about my Wi-Fi here in Cabo, so I'm going over my mic. I am, but I'm very excited to be here and hearing this. So, yeah, it's just about student outcomes. And have you seen any student outcomes improved since integrating this? And then how are you able to capture, are you able to capture, how are you able to capture that data?

And so I think that this is going to, we don't have enough data at this point. I think students just took the midterm, and so it's difficult to say. I think the data that we do have is all subjective. It's students telling us, oh, this was a great answer. This was extremely useful. Or like, oh, this wasn't the correct computation and things like that. So we can use those things to improve the bot from our perspective. But as we move forward, I know that one of the courses that is interested in a chatbot in quarter two coming up is very interested in tracking outcomes and actually taking a scientific approach to being able to measure that impact. So I'm very excited to work with that class, and we'll see if that's moving the needle. Right now, we have just subjectively, it sounds like it's useful, which usually means it's saving time, and it's adding value subjectively.

Thank you, Marissa. There's also going to be an anonymity issue there, right, Brandon? I mean, if we want to ensure student anonymity, then that becomes pretty hard to link to outcomes. So I guess there's going to be something to navigate there when the time comes. Hendrik, maybe we'll bring Sia back on to help us answer some of these questions now. And Brandon, we will see you soon. Don't leave.

Hi, Sia. I just wanted you to be present for the Q&A because some of these you might actually have a little bit to say about. So Andy, Oliva, are you still around? We'd love to hear your question. I think this is more for Richard. Hi, Andy. Can you hear me?

Yes, we can hear you.

Yes. Awesome. So, I was just curious, in using those Stackbots, were there any plans to integrate them into new students that were looking at courses in the college and helping them, you know, pick a curriculum and how they intersect with each other and maybe the why behind what each course is? I remember when I first was going to college, that was a very overwhelming process was picking classes. And I think that a lot of these things will naturally evolve.

So, the tool that I told you about, this so-called Wembley Insights, which was for faculty to learn about one another's courses. I mean, if that's in place, then there's nothing that stops a student potentially having some access to some version of that and putting in a query like, you know, could you create a path? I want to be an investment banker and could you create a path through a set of courses for me to help me, you know, in that process? And if we've got all of those classes somehow sitting in our vector database, then presumably that's going to get an answer. And so, I mean, I just think, you know, that once we have all that data in place, it's just going to generate all sorts of uses that we might not have originally thought about. But that would be a very small change from having a tool for the faculty to look at one another's courses. So, it's not, we're certainly not there yet, but it seems absolutely like one of the tools that one would want to offer to students.

Andy, I just dropped the name, a name of one of our members in the chat, Samar Abedrabo. She is a community college professor of biological sciences in the East Bay, and she's actually working on actively building this now, like an academic advisor, a counselor for biological sciences in her community college district. So please DM her. You guys potentially could collaborate.

Sia, I'm curious if that's a use case that you hear often as you're working with higher ed faculty and admin, like an administrator, counselor, GPT.

We're definitely hearing that more. I think it started first with professors and now it's evolving to staff that's helping support students and the different types of solutions for them. And what we're finding super interesting is universities are uploading their proprietary data and training their custom GPTs on that to help students be able to solve these kinds of challenges. And I think that becomes really powerful for universities that have a significant amount of data to help students. Thank you, Sia. Thank you so much for the question, Andy.

Okay. John Burns, are you still here? If you are, your questions received some thumbs up if you'd like to ask Richard or Sia a question.

Hi, John. Yeah, thank you. And I just wanted to say, hey, good to see you. This is wonderful, by the way, just like all the organization and the people speaking. This is just fantastic, like just to connect. Yeah, I'm just curious, like given that you've ingested all these lectures, slides, like has there been any impact on practice in terms of your lecturers or your facilitators or your teachers? Are they looking at it and going, oh, wow, I didn't realize you asked so many questions in your session. Or I didn't realize the style of questions you asked, or I didn't realize how many slides you used. Like that's always the tricky part about teaching is you don't get to see how other people operate. And I just find that interesting.

Right. I mean, at this point in time, we don't have a huge amount of visibility, but this tour, this Wembley Insights I keep referring to, I mean, it's going to let one faculty member look at another faculty member's class. So I mean, we will be able to learn from one another in that sense. I mean, one thing that I found once I started getting material to be ingested, it made me think much more carefully about how I would organize it. I've actually, this term, for the first time in many, many years, gone back to my notes and actually organized them, knowing that something is going to go through them, looking for various parts, you know, here, there and everywhere. And so in some fashion, they're better organized than they were before, because it became apparent that they, you know, if I'd ask it to summarize some part of the lecture and that didn't show up, it was usually because I hadn't put in an appropriate heading or a title or a section identifier or something like that. So that was just a kind of weird thing that came out of the process, because, you know, if I'm honest, my notes were pretty static for about 10 years. And even though the content hasn't really changed, the structure of them suddenly did to make themselves amenable for this sort of querying that was going to go on that I realized was not going to start. So all these kind of strange things that happen that I never, you know, anticipated. You know, I always thought of the tour as being a tour for helping students learn the material, but we're seeing these queries like, where is the midterm going to be? And I might well have said that in class, and they're almost using it for, you know, that intuitive side of class as well, as I didn't think anyone would ask me that. But, you know, it's going to be fascinating to see all the different sorts of ways that people were using it that we hadn't anticipated.

Awesome question, John, and so good to see you. Hope to see you again soon.

Sia, on that note, you speak with a lot of faculty members. Are you finding that part of educating higher ed professionals, faculty, admins, on adoption has to do with what Dr. Waterman's describing, like optimizing the functionality or the outputs of the chatbot has to do with the way they're organizing the data, the inputs?

Right now it does. We hope that it's going to get better in the future and that faculty won't have to spend as much time organizing and that we'll be able to parse through it. But yes, for right now, the best outcomes come with the way that Professor Waterman described it.

And the next question, this is interesting, it's a little bit different topic, but if we could please spotlight Jordan. Jordan has a really interesting question about how universities are communicating these initiatives in AI adoption, like in the public realm.

Hi, Jordan, welcome. Oh, you're muted, Jordan, there you go. Let's see if this works. Can you hear me now? Great. Awesome. Oh, so this sounds fantastic that y'all did this. I just wanted to know about like, how did you get the word out? Do you have like a comms team or how did you stay on for messaging? Because before I left Salesforce, we launched a couple projects that were powered by open AI and having comms lean in to constantly stay on message did a lot for user engagement. So thanks.

Do you want me to ask that one?

Yeah, Richard, did you get support from PR, comms? We don't really need it because you're the professor in the classroom. It's a very different use case than a business case. But I mean, I stand up in front of my 110 students, I say, you know, here it is, here's where you find it. And then Nupur and Saran evangelize for it. And we're kind of good to go. So that side of things was not a difficult thing for us. Because I'm standing in the classroom every week talking to the consumers of this, that wasn't a hard part specifically.

Nice. What about your peers? Did you have other faculty? Like you kind of had to sell on this at all or were you just focusing mostly on your own classroom?

Well, I mean, I developed it, you know, obviously for my own classroom, but I'm, you know, because I'm interacting with the faculty a lot. I mean, they know what is going on. And as Brandon was saying, I mean, some of the other ones are starting to pick it up now. I mean, they know it exists. And I think a lot of this is, to be honest, because I think there's going to be such a demand will be quite organic. I mean, it will just, the demand will appear as through word of mouth more than anything.

everything else, and the student requests and expectations. So I don't feel that's going to be our hardest part, to be honest, but maybe that's because education is a little different than the pharmaceutical industry or the finance industry. We have a captive audience, put it like that, in the classroom. Thank you. Thank you, Jordan. Good to see you.

Okay, friends. That's a wrap. We did it. I think this is definitely one of my favorite sessions that we've hosted in a year and a half. See you, Dr. Waterman. Thank you so much for bringing us all together. That was really fascinating. We can continue the conversation in the chat afterwards if anybody's interested.

Dr. Waterman, I hope we can reconnect maybe six months from now, see how your project is going. What the auction is like, yeah. It's going to be amazing, and we're really excited to keep supporting you folks. And please do invite your students, invite the other faculty members. We are really happy to host them here. We're all learning together.

In fact, on the 24th, two days from now, we host two enablement sessions a month here. They're very small groups, just like this. And we have solutions engineers, solutions architects, other software engineers from OpenAI. They show up after community members fill out a survey, and they teach us how to solve the problems that community members have surfaced as being top of mind. So whether it's how to approach from an architectural standpoint, how to get an initiative started like Dr. Waterman did, like where do you start? Some of them are more technical for folks that are actually building with our API. If you read the summary of the webinar, you can learn a little more about it then, but we are pleased to invite you to that later this week.

And Dr. Waterman, anybody in your sphere that's interested in learning a little bit more, they're welcome to join our community technical office hours. And then also, just want to remind everybody that as our season is coming to a close, that we love your referrals. This is right now an invite-only community. It's the OpenAI team, it's our OpenAI forum members. Those are the folks that we invite into the community, so we'd love to hear from you. And please circulate the, basically it's a VIP referral app, and we're all learning from you and building upon the network effects that we've built here.

Last but not least, I have a very exciting announcement. We haven't published it yet, but the new VP of Research at OpenAI, Mark Chen, he's been with us for a very long time, but newly appointed VP of Research. And Terence Tao, one of our favorite mathematicians, are going to be hanging out with us in the forum on December 3rd for a virtual talk on the future of math, now that AI is here and our O1 model has been released. So you guys are all welcome to register for that as well. We'll be sending out a newsletter soon, so you can early register.

And then Caitlin and I, my colleague in the forum, we are working on hosting one last in-person event to close out the year. You should be seeing that soon too. Thank you, Dr. Waterman. Thank you, Sia. Thank you everyone for being here tonight. Thank you, Brandon. Thank you, Nipur. Thank you, Chetan. Thank you for all of our community members. We had a beautiful crowd tonight. It is such an honor to host you. Cesari, I saw you too. Good to see you.

I hope you guys have a wonderful night and we will see you soon.