Sign in or Join the community to continue

Event Replay: Polling Workshop: AI Tools for Election Surveys

Posted Oct 08, 2026 | Views 25

Share

Speakers

Lucas Stockslader

Global Affairs Insights @ OpenAI

Lucas Stockslader leads primary research for Global Affairs Insights at OpenAI. His work draws on experience in data, analytics, and research across politics and technology, helping organizations understand audiences and use evidence to guide decisions.

+ Read More

Jon Cohen

Founder and CEO @ TrueDot

Jon Cohen is founder and CEO of TrueDot, an AI-powered insights platform based in Palo Alto. He is a seasoned researcher and senior technology executive with decades of high-stakes survey experience. He ran presidential and midterm campaign polling at The Washington Post and ABC News, consulted with the NBC News Decision Desk, and currently advises Decision Desk HQ. In his previous role as SurveyMonkey’s first and only chief research officer, he delivered significant revenue gains over nine years of building next-generation tools and consulting on survey technology across Silicon Valley and globally. He was named “new school pollster,” changing the game in Washington by the National Journal. He has co-authored publications on Californians’ opinions on their elected officials for the Public Policy Institute of California and he currently serves on the PPIC Statewide Survey Advisory Committee. He received his BA from Johns Hopkins University and an MA in political science from the University of California, Berkeley.

+ Read More

Mark Murray

Editorial Director @ OpenAI

Mark Murray is Editorial Director at OpenAI, bringing nearly three decades of experience covering American politics as a veteran editor, reporter, and storyteller. Before joining OpenAI, he spent 21 years at NBC News, where he served as Senior Political Editor. In that role, he directed the network’s political coverage, managed NBC’s Political Unit, oversaw its extensive polling operation, and wrote the lead stories on election results and public opinion trends.

A trusted voice in political journalism, Mark has reported on every U.S. presidential election since 2000, appearing regularly on television, radio, podcasts, and digital platforms to break down polling data and explain the dynamics shaping American politics. Earlier in his career, he served as Deputy Political Director, Off-Air Political Reporter, and writer at National Journal, where he covered Congress, immigration, labor, and education policy.

Throughout his 27-year career, Mark has been recognized for his ability to translate complex political developments into clear, accessible narratives. Known as a collaborative leader and skilled communicator, he has guided teams of reporters and researchers while providing audiences with thoughtful analysis across media.

+ Read More

Brent Buchanan

Founder & CEO @ Cygnal

Brent Buchanan is an international pollster, messaging strategist, and author based in Washington, DC. Buchanan, who was recognized by the American Association of Political Consultants as a “40 Under 40” and by Campaigns & Elections as a “Rising Star,” has served in numerous polling, comms, and strategy roles for governors, presidents, legislative leaders, and major corporations. He serves on the board of the International Association of Political Consultants and was formerly its treasurer. Brent is Founder & CEO of Cygnal, an award-winning international polling, public opinion, and predictive analytics firm also based in Washington, DC, whose clients include Fortune Global 500 companies, foreign heads of state, presidential campaigns, U.S. senators, congressmen, dozens of state legislative caucuses, and trade associations. The company was recognized by Inc. 5000 as the fastest growing private research company in the U.S. and in the top 15% of growth for all private American companies.

+ Read More

Ilana Ron-Levey

Managing Director @ Gallup

Ilana Ron-Levey is a managing director at Gallup and a member of its executive leadership team. She oversees talent, client engagement and financial performance for Gallup’s Global Analytics division and directs researchers working on public-release studies and social impact partnerships, including the Gallup World Poll. Her research spans survey methods, program evaluation and global public health. Before Gallup, she co-directed Abt Associates’ qualitative methods center and led international research and evaluation projects.

+ Read More

Michelle Jaconi

Consultant @ OpenAI

SUMMARY

This workshop brings together pollsters to explore public attitudes toward AI and how the technology is changing the way they gather, analyze, and interpret public opinion. Lucas Stockslader presents OpenAI’s AI Sentiment Tracker and explains how question wording and survey populations affect measures of AI sentiment. Jon Cohen demonstrates TrueDot and describes how he uses ChatGPT and Codex to query polling data and speed up research workflows. In a discussion moderated by Mark Murray, Brent Buchanan and Ilana Ron-Levey examine public attitudes toward AI, Gallup’s AI phone interviewing experiments, and the importance of survey quality, transparency, and trust. Across the discussion, speakers emphasized that better tools still depend on quality responses, sound methods, and transparency to earn public trust.

+ Read More

CONTENT & TRANSCRIPT

00:00:00 | Michelle Jaconi

I'm Michelle Jaconi and I wanna welcome you to the DC workshop. This is not an office. It's actually called a workshop and I'm a lover of words, so I wanted to stop and think about that for a moment. I love the feel of this room. It kind of feels like a boutique hotel crossed with a high tech classroom and that's exactly what it is. I hope this is the first of many times that you come because the whole point of this space, it was designed to invite you into the conversation around artificial intelligence and invite you to shape that with us and share your knowledge. This room is a wow. It's a wow to me. I see so many friends.

00:00:35 | Michelle Jaconi

It feels like a birthday party and a nerd convention and I love it, okay? So thank you so much. One of the things that I have two goals, I like to be really explicit about what my goals are for you today. One is for you to learn something new, okay? It could just be one thing that makes you level up a little bit, like a little bit faster at something you do. You're gonna do more than that, but let's just start with there, okay? Two, I want you to meet someone new and I don't mean just shaking your hand, I'm Michelle, I work at OpenAI.

00:01:03 | Michelle Jaconi

I want you to really think of someone that you learned something from here today so you can just put that note in your head to follow their career. The people sitting next to you and the people presenting today are pretty impressive, okay? This is a very highly curated audience. We had to close this guest list. A lot of people were lobbying. It felt like a really fun thing to get in this morning and so the fact that you're here is that one of us really vouched for your intelligence and thought you would learn from the presentations today, okay? One, I think that, I'm just gonna give you a little preview.

00:01:36 | Michelle Jaconi

You are gonna hear from somebody who was a Washington Post polling director and then ditched us for Silicon Valley, okay? He went to head up research at SurveyMonkey and then started his own company. This is somebody you can learn from. We all can learn from. Two, you're gonna meet somebody from Gallup, an organization that has been innovating for a really long time and in fact, bought this building to show the power of public opinion research, somebody we can learn from, right? But we're gonna start with and end with something absolutely new.

00:02:12 | Michelle Jaconi

The final panel is going to also feature a CEO of a survey company who brought new data that he put in the field just to share with us today. Okay, love that, right? So we're breaking some news. I'm gonna start with a colleague of mine who has two of my favorite human traits mixed together, like complete political genius and a diehard fandom of the Buffalo Bills, okay. And so please, Lucas, you wanna come up?

00:02:45 | Lucas Stockslader

Thanks, Michelle. It's a hell of an intro. Hi, everyone. It's really nice to see everyone here. Really happy to see all of your faces. And like Michelle said, I definitely don't, hope that this is the last time that we see you all here. My name's Lucas. I lead our political research program here at OpenAI. I'm gonna go through a few things here, which is one, just our current state of the AI sentiment landscape.

00:03:08 | Lucas Stockslader

All the things that we're seeing here, some of the tools that we use to really understand the landscape, and then some of the nuances and some of the things that we think about when we're starting to think about how AI is being polled and how we can start to track some of these things. The first thing that I wanna talk about is the dichotomy between two different squares that we're trying to circle here, which is we hear about AI a ton. It's everywhere.

00:03:33 | Lucas Stockslader

And a lot of times, we see this when we go and do qualitative research, which is we have these wide-ranging conversations about AI, all the influences in people's lives, people's worries, people's fears, people's hopes for the future. And oftentimes, we'll get to the end of these conversations and we'll say, and how is AI gonna affect your vote? And we'll get blank stares. Or we'll say, AI's not really gonna affect my vote, but there's more important things going on. And it doesn't really feel like there's more important things going on, especially here in D.C. oftentimes. And we see this replicated in a lot of our polling, which is we see really high concern about AI's effect on society.

00:04:06 | Lucas Stockslader

We also say about half the voters will say that AI's gonna affect their vote in some good way or another. But only 4% of people are really choosing AI as one of their top voting issues. So how do we really think about these things and how come AI hasn't become a dominant political problem yet in this election?

00:04:23 | Lucas Stockslader

The first thing that we have to do is take a look at these first two numbers, which is even though we see really high concern and really high worry about AI as a society, we start to see that splinter really, really quickly when we go to the personal level, which is once we ask people about concerns for AI's on their personal life, we see that already drop around 24 points down to about half of people who say that they're concerned.

00:04:43 | Lucas Stockslader

And we see this replicated in open ends as well, to really go back to our qualitative studies, which is people answer their concerns about AI's in a lot of different ways, and a lot of different ways that you ask it. People are much more likely to say their, to talk about their convenience and their productivity when we let them talk at us about their own personal lives. But people are much more likely to talk about job losses, much more likely to talk about things like skills loss and independence when they talk about society as a whole.

00:05:11 | Lucas Stockslader

So we're already seeing that the way that we ask these questions can be really, really different in terms of the answers that we get. But we have to go even further, which is where is this energy in the political system coming from? And we can see that among people who are strong optimists and people who are strong pessimists. Optimists are much more likely to say that AI is gonna affect their vote in about a month than pessimists. This energy is coming from both sides, but it's actually more likely to come from optimists than it is from pessimists. Most of these people fit in this movable middle here.

00:05:40 | Lucas Stockslader

And we see this when we come into focus groups as well too, which is a lot of people walk into the room somewhat skeptical, and that familiarity starts to breed comfort, but people stay in that movable middle for a really, really long time. We doubled down here, and we're gonna keep on doubling down as we go through this because optimists are even more confident in their assessment of AI's effects on society. We see this in a lot of the different polling in a lot of the data that we look at.

00:06:06 | Lucas Stockslader

And we can see it online too, which is the AI optimists out there are extremely optimistic, and they're extremely confident, much more confident than pessimists that we see out there. And so we see this default position starting to form, especially when we think about people that walk into rooms skeptical. And we can see this lastly, as we go through, whereas oftentimes we see independents kind of fit in between Republicans and Democrats, we see that the default position for independents is actually to be more negative than Republicans and Democrats on AI's effect on society.

00:06:38 | Lucas Stockslader

And as we go through and really understand kind of like what is the political problem that we're dealing with here, we can see that Republicans start pretty positive, but then everyone kind of fits towards this really skepticism about AI. The next thing that we have to do is really understand how AI is gonna splinter when we start to ask some of these questions. So if only 4% of voters say that AI is gonna be their top issue, it's actually the wrong question to ask. Instead, we can see that AI is actually gonna affect people's vote in a myriad of different ways.

00:07:10 | Lucas Stockslader

When we ask people when deciding to vote, how much, if at all, does AI factor into your views on each of the following? We can say that AI actually has a really big factor in a lot of these things, everything from reliable information to jobs and to wages, everything down to healthcare and democracy. And this is the kind of big question that we're gonna end with today, which is how you ask these questions can really matter the types of responses that you can get.

00:07:34 | Lucas Stockslader

So even though 4% choose AI as their top voting issue, over the next few years, we can say that AI's gonna have a massive effect on public opinion, especially when we look towards the next political environment. We know that things like healthcare costs, everyday costs, are gonna have a massive swing in this next election. And we can see that people are actually split down the middle pretty well in between what impact people think that AI is gonna have on all of these things.

00:07:57 | Lucas Stockslader

So when we talk about the types of things and the way that we ask a lot of these questions, we think about AI and you'll hear opinions saying, AI is fire, AI is electricity, AI is the next technological revolution. Polling about AI is like polling about the printing press in 1441. This just happened. And we're just starting to understand the nuances between these things and how to ask about these questions. And just to make sure that we're showing them that we did our homework, this is a question from 1987, shout out to our friends at the Roper Center who are collecting all these data and housing it so expertly.

00:08:36 | Lucas Stockslader

Back in 1987, when people asked, if AI is gonna happen, what do you think the effects are gonna be? People always thought this was a bad idea. People were really skeptical about AI. And we, at 40 years later, have seen very little movement. We're highly likely over the next three years to see public opinion changing on this more than it's changed in the last 40 years. So when we think about these, it's not enough to ask questions about, is AI gonna affect your vote, how is AI gonna affect your vote, we can't really compare AI to a lot of the things that we ask about AI now, whether it's gas prices, healthcare costs, et cetera.

00:09:12 | Lucas Stockslader

AI is likely to splinter as we go through these things and affect different issues in different ways, especially as the next revolution comes up, and especially as it starts to affect different industries and different issues in different ways. The very last thing I wanna say is we're tracking these all the time. A lot of this data is from our friends at TrueDot who have just become awesome thought partners in a lot of the ways that we think about these things and a lot of way that we measure things. We publish an AI sentiment tracker that's 100% Codex built and 100% automated at this point.

00:09:45 | Lucas Stockslader

And what it does is it tracks and collects binary outcome information from high quality public pollsters. Some of your polls are in this tracker. Everything from Marist, Pew to Gallup. And we can really start to see some of the nuances between how we ask these questions and who we ask them among, which is likely voters still generally approve about AI. But when you ask things about, do the risks outweigh the benefits? Does AI do more harm than good? And then depending on whether we ask them among likely voters or the general public, we get wildly different answers.

00:10:17 | Lucas Stockslader

So I'd encourage you over the next month, as we all go and do our post election autopsies and really understand how we're thinking about these and how we go into 2027, it's not enough to just ask one question about AI as we go through this. We really need to dig into these, all of these different nuances really, really clearly. The very last thing I'll say about this is this new data that you'll see next sometime this week, if you subscribe to the prompt, our substack that Mark Murray is just doing the Lord's work on, really houses a lot of these really interesting and awesome information. With that, I'm happy to answer any quick questions.

00:10:54 | Lucas Stockslader

I might've gone over for a little bit. And I know that we'll probably have plenty of time to go through this just a little bit later, but happy to answer any questions about our work here, some of the information, some of the data that we showed, and also happy to pass the mic on.

00:11:12 | Michelle Jaconi

If you're feeling shy, there'll be networking time at the end to bother Lucas.

00:11:14 | Lucas Stockslader

Thanks guys.

00:11:21 | Michelle Jaconi

Okay, pop quiz time. Sorry, Professor Owen, I get to turn the tables on you. I want you all to think about how many active users does ChatGPT have in a week. Okay? You have option A, B, C, and D. I want you to think about that, and I'm going to bring up my colleague and dear friend, Mark Murray, pride of UT, to give you the answer. Thank you.

00:11:49 | Mark Murray

Thank you for that, Michelle. Lucas already ended up having a little prompt of the prompt newsletter that we ended up having. If you are a subscriber, you might know the answer to this, but for those who don't, the answer is 1.2 billion with the B. Thank you for that little trivia, Michelle. As Michelle mentioned, I'm Mark Murray. I'm the Editorial Director here at OpenAI. But before coming over to OpenAI, I spent more than 20 years as a political journalist at NBC News. And one of my responsibilities was managing our polling operation. I worked with many of you here in the audience. Some of you are our vendors and pollsters. Some of you were my good sources.

00:12:34 | Mark Murray

Others, I ended up getting your emails and putting out the poll releases. But in my current capacity at OpenAI, I always had this kind of question. I'm like, how would I have used these tools when I was a political journalist in dealing with polling information? This is something I think about all the time. And one of the cool hacks that I currently end up having and what I end up doing is we're kind of crunching a lot of the public polling that comes out with AI, helping to put it into Lucas's automated Codex AI sentiment tracker, is I ask Codex and ChatGPT work to create a weekly compendium of all new public polls that come out.

00:13:14 | Mark Murray

It gets sent into my inbox every Friday, produces all the links, all the, gives me the crosstabs, gives me the top lines, gives me the summaries. And then when I have a little free time on Fridays, which I always hope, I get to spend an hour or two going through all those results. And it is one little way how I've been able to automate my own work using ChatGPT and Codex. You know, it used to be when I was getting polls, people had to send it to me, I had to scour through Twitter, I'd have to have Google Alerts, and now Codex and ChatGPT work just put it all in my inbox.

00:13:48 | Mark Murray

And the quality of work has gotten so much better about a year ago. Sometimes when I was doing these things, it was, some polls could be a year or two old. It was outdated. That's not the case at all anymore. And according to analysis that we ended up doing a privacy-preserving analysis, we actually find that tens of thousands of people every single day use ChatGPT for polling information. And whether it's like getting the polling information that I am trying to do, asking questions about methodology, trying to actually see who's up, who's down in a particular Senate race, people are using ChatGPT.

00:14:26 | Mark Murray

But one person who's using our tools more than anyone else, more than I'm able to do, and one of the people who actually inspired today's talk is Jon Cohen. As Michelle introduced Jon, He was the former director of research, of the Washington Post poll. He then went over to SurveyMonkey and now he's the founder of TrueDot.ai, the polling operator and who is using AI in an extraordinary ways. And, Jon, I want to turn the mic over to you to do an awesome demo.

00:15:00 | Jon Cohen

Thank you, Mark. It's really great to be with all of you today. Some familiar faces. I look forward to meeting the rest of you after this. And I'm in Washington all week so I'm happy to be back here. I'm here to talk about AI. I'm glad to see there are 1.2 billion active. I may be the most active among the pollsters. I'm not quite sure. But it's transformed the way I work and those of you who haven't already brought it into your work flow, I encourage you to try it and we can talk at the end. I think it's never been a better time to be a pollster in part because of these tools.

00:15:35 | Jon Cohen

I think the change is a good thing. I'm going to walk through some slides and time willing I will have a demo for you as well. Live demo. This is exactly 20 years ago to the day I walked into the Washington Post newsroom as polling director. The Post thought it would be a good idea to replace two veteran pollsters with one seven weeks and one day before the 2006 midterms. It was not a great idea. I worked 75 straight days, probably 15, 16 more hours a day. It was incredibly intense. The newsroom looked a lot like this. There was no movie star, there were no typewriters, but we did all of our polls exclusively on landline in those years.

00:16:14 | Jon Cohen

It was super fun, it was really intense. We did a lot of work. I wrote more than 300 polling stories in my nearly eight years at the Post, including more than 100 that made it to the front page. Those of you who still know what that is. It was really fun, but it was never not intense. One of the reasons it was so intense is I would arrive in the newsroom at 5 o'clock in the morning. The data would be there. By 9 AM, I had to tell the editors what the real stories were. What they should be putting on that front page. What they should be printing. And because it was 2006, we were putting it online at 5PM.

00:16:52 | Jon Cohen

So, it was really intense. These were what we had to work with. If you see, there is a pollster in the room today still pouring over paper crosstabs. Thank you Lee. It is great. It is still the work. The work is still, at that time, I got sent, maybe by Evans, a 400-page PDF with 24 columns across each page to go scour, and a trend report to look through, and SPSS. Sorry to my friends at IBM, but I have been on a career-long crusade about never having anyone I care about have to use SPSS. But at the time, I had it, and these were the tools that I had to work with.

00:17:36 | Jon Cohen

So we'd write the story, we'd publish it at 5 o'clock, and at 5.01, my phone would ring. And for me, in 2006, it was literally every time we put it on a poll, it was Ramen Manufacturer. As Nancy Pelosi, one of her top deputies at the time, his job, as he saw it, was to know more, just know more. He hadn't read the story. His aides probably hadn't read the story by the time he picked up the phone, but he wanted more information.

00:18:00 | Jon Cohen

And I was stuck with the same tools, the same static crosstabs, the same SPSS to answer Rahm’s questions, which were probably screamed at me on the phone, and I'm holding it out here and trying to answer it. I wish I had the tools that we have today. I wish I had ChatGPT. I wish I had Codex. I wish I had the tool that we built on top of these foundation models, where you can have all of your data in one place and be able to query it just using English. No matter what decibel it was coming at you. We could translate it. We could answer those questions.

00:18:40 | Jon Cohen

The difference today than even six months ago or a year, like the numbers are right. Having a system where you can query in natural language and be 100% confident that the numbers are right is amazing. I can't tell you the number of times I shot awake at 3 o'clock in the morning wondering if the wrong number had been printed in that newspaper. And then I had to wait a few hours for it to be delivered and checked. And almost always it was right. But that was scary. I wish I had these tools. So TrueDot Connect exists within ChatGPT.

00:19:10 | Jon Cohen

It's a plug in for those of you who haven't used those, which are combinations of connectors and skills that you can have that make things look the way you want them to look. Within ChatGPT, you can invoke TrueDot and the numbers come in and you can ask a question like this, looking at the latest results, what's the top story and what changed the most? And the numbers on the next screen are 100% accurate. These systems are incredible. ChatGPT is brilliant. OpenAI is brilliant. And it's random. There's a stochastic quality to this. You all know that. And so it's not necessarily reliable as a system, but you can build reliable systems and trustworthy systems on top of these brilliant technologies.

00:19:59 | Jon Cohen

And that's what we've done. So we run a weekly omnibus poll. I just put in the exact prompt you saw and you get the answers. You know what's at a high point, what's at a low point, where things have moved. You get the first story. Now this isn't necessarily the story you would pitch or push, but having a really solid, accurate first pass allows you to dig deeper. You're compressing the workflow into something that's manageable and you can follow your curiosity and really dig deeper. But it's not just about compressing the workflow. It's about dramatically expanding the context.

00:20:34 | Jon Cohen

When I was in graduate school, one of my jobs was taking all the old Merv Field’s, old California polls, and digitizing them. They lived in binders. Just like, I should say, the Trend Reports lived in binders when I showed up at the post. There's a lot of binders in my life. But California polls lived in binders. We coded them one by one by hand. I've now, with Codex and TrueDot, created a California aggregate file of more than a million cases from 1956 to 2026. I have a hard time pulling myself away from the dataset. I got my start in California surveys, and I love looking at this. It's a 70-year trend. What can you do with a 70-year trend?

00:21:17 | Jon Cohen

I asked TrueDot, what's the single biggest thing in this entire dataset across this whole repository? And the number one thing it came back with was the shift in the educational components of the two parties. From the 60s, late 50s, through the 80s, more highly-educated people in California were more apt to be Republican than their counterparts with less formal education. That's switched. Now you have a situation, which we all know, where Democrats, at least in California, I think it's true nationally, are more likely to be Democratic than people with less formal education. But that's the kind of thing you can do with 70 years. You can also get rather geeky, as maybe some of you are, like me. I don't know.

00:22:03 | Jon Cohen

But how have weights changed over time? Yeah, so we have a million cases. How have the weights changed? And you can see, for many decades, the weights on education and the weights by age basically were one. You can understand the resistance of many pollsters to not do weighting, certainly weighting by education, until the 2016 AAPOR report came out and said you really need to be doing this by education. The data show why there could have been some resistance. It was really kind of a nice-to-have for multiple decades, and it became imperative that you do statistical weighting to make sure that the populations that we're representing are, in fact, more representative.

00:22:45 | Jon Cohen

You can see that really change with a data set like this. Go back 20 years. After I made it through the 2006 election, I convinced my bosses to hire someone else. I had a new challenge, and that was that averages had come online. I wrote in the post that averages weren't necessarily a great thing for our profession. I probably lost that argument. I definitely lost that argument. I did argue for standards and transparency, and that's where I've kind of hung my hat for all this time. That was what we talked about 20 years ago.

00:23:16 | Jon Cohen

But now, if you have all the data, as we do, as you have access to, as you heard Mark gets a report, you can run your own averages. I built an app using Codex that kind of has advanced averages across every state. It's updated every day. It gives a little readout. Marist put out a Maine poll this morning. That's immediately here, with Trend, compared to all the other Maine polls, and there. And so you can build things like this, or you can build something in your own workflow to do. A tool like TrueDot allows you to kind of have a trustworthy system for all the data, raw and crosstabs. But you have the core technology is the amazing OpenAI models.

00:23:57 | Jon Cohen

And so thank you to my host here. But it has to be right. It took me two and a half years to roll out TrueDot Connect because, you know, I come from Silicon Valley, where people just roll out stuff all the time. I have a certain bar as a pollster, spent my career in it, I know you do too, where it has to be right. And to build a system that's trustworthy on top of these brilliant technologies takes effort and a lot of care and a new data model. And so we've done that. Please, you know, kind of hit this QR code if you want to learn more.

00:24:26 | Jon Cohen

It needs to be right in our hands when we run it, but it needs to be right in yours as well. So I don't know how much time I have, but I can switch computers, answer a question and... How do I go back?

00:24:39 | Mark Murray

Jon, thank you for that. That was amazing. Jon, stick around. I'm going to throw a couple of questions to you, Jon, then we'll open it up to the audience. The one question I have, and I think there are some people here who are in the audience, who don't use AI all that much. And so, what advice would you give them for somebody who is kind of dabbling for the first time on kind of getting comfortable and incorporating in your own work?

00:25:04 | Jon Cohen

Thank you. First of all, just start. How many people have started using AI? Most people. So, I would just do it. Right? It's incredibly easy. The tools get easier over time. You guys rolled out a new thing called Dot. Thank you for the homage to TrueDot. I don't think it had anything to do with that, but I do appreciate the connection, where it's just a lot easier to start. And so, I think you should. I think you should also, kind of, your expertise is gonna kind of convince you, hopefully, over time, that you're producing good content. Right? Anyone can produce numbers using these tools. You're gonna trust it over time when you see your data reflected back at you accurately.

00:25:49 | Jon Cohen

And so, kind of, like, bring your expertise. Also, really important to iterate. You might not get it right the first time, but keep at it. Kind of, particularly with the new concept of the Dot, it learns. And, so start, keep going, and I would say have some fun, because I think it can be fun. I mean, I had fun in the newsroom, like, not every day, but most days. And I think this can be really fun working with AI. So...

00:26:13 | Mark Murray

Jon, my other question is, being a small business.

00:26:16 | Jon Cohen

Yep.

00:26:17 | Mark Murray

Not having a whole lot of people working with you over at TrueDot. Talk about how AI actually helps in that regard.

00:26:25 | Jon Cohen

Well, I mean, it's changed the workflow, not just for the polling use case that we talked about, but for engineering. You know, we can now talk about a feature we want one day, and it's real the next day. It used to be, I started this company three years ago, AI existed, but nothing like it is today. You needed more engineers. You needed more time. You had to commit to a UI. One of the things that we bet on was that you could do everything through chat. That was a bigger bet three years ago because it took a lot of time to build things. It's not a big bet now. It's clearly what people want.

00:27:00 | Jon Cohen

And so you can do that, and it's just the tools are incredible. What I have on screen here, just so you notice, so Marist again put out a Maine poll. I just said at TrueDot, what’s new in Maine? And it tells you some stuff. So does anyone have another question you wanna ask? So I should say this repository has every single Senate poll that's been released publicly this cycle. So you can do things like, and these are just things that I did before, compare the gender splits in every poll that's been released and sort it by the gender gap.

00:27:34 | Jon Cohen

I had my California one, so I said did California party ID over time for as long as it exists by race. And you can see how these start to change. So again, this is looking at a million cases. But does it, fire off a question. If anyone's brave,

00:27:51 | Mark Murray

Any questions?

00:27:53 | Jon Cohen

can be about any Senate poll this year. Yeah, Margaret, thank you. How is the Hispanic vote splitting? Do you just wanna let AI choose which is most important?

00:28:10 | Audience member (Margaret; surname unconfirmed)

Yeah. Yeah.

00:28:13 | Jon Cohen

Okay. Don't have to use a question mark. You don't have to spell correctly. You know, you can just go for it. So let's see what it says. Okay, so it came back and asked a question. Margaret, which one was?

00:28:35 | Audience member (Margaret; surname unconfirmed)

Texas, Ohio, and Florida.

00:28:38 | Jon Cohen

Texas, Ohio, and Florida. Okay. So, this is why I had slides, because demos sometimes happen super fast. Sometimes, you know, kind of here it goes. So. Again, these are all, so if, there's no raw data on the Senate questions, I mean, except for ones that we've conducted for clients. But those aren't in this repository. This is everything that's been published.

00:29:14 | Mark Murray

Jon, as it's thinking, I have one quick question, and then we can turn more to the audience once Margaret gets her answer back. And that is, when Lucas had his presentation, he ended up throwing that Roper poll from 1987, and the data, when we reran it using TrueDot polling, and the numbers were very, very similar. Kind of put on your analytic hat as a pollster and kind of make sense of what that result showed and what that kind of means going forward for AI.

00:29:42 | Jon Cohen

Well, first of all, I love that Lucas found that question. I love you kind of asking questions that were asked many decades ago and rechecking them. And so thank you for that. I think always surprised when things come back as close as they did, given all the many, many changes politically, sociologically, technologically in the last 40 years, I thought it was rather incredible. The next step for us is to dig into the cross tabs, we'll see how we'll kind of pull the raw data from Roper and compare it to ours, and I think that would be a lot of fun, but it's really remarkable. And I think Luke has said it, we're just at the beginning of this, right?

00:30:19 | Jon Cohen

We're so focused on the overwhelming, kind of negativity in some of the polls about kind of what's new, we don't know how much of that is real or lasting, it's a moment, there's a lot that goes into that, a lot of work that we've done that we will probably put out academically, does it matter when you're kind of nudging on uncertainty? How much does knowledge matter? Because there are a lot of us to look at, but we'll start with the cross tabs and the data.

00:30:46 | Mark Murray

Looks like we got our answer.

00:30:51 | Jon Cohen

So again, a good system has to also highlight what the caveats, right? So, you know, this is, it chose to list the amount of sample, right? That it matters. And it matters that the Rasmussen poll is not published, and you could choose to throw that out. But kind of, again, part of, one of the concepts of having a trustworthy system is it could say no, when you ask for something that isn't a real comparison, and that it'll tell you kind of the important things to know about each of those pieces. And then you can say, put it in a chart. You can say, put it in a chart with an OpenAI logo.

00:31:27 | Mark Murray

Rob, you have a question? There's a mic coming right here.

00:31:33 | Audience member (Rob; surname unconfirmed)

Thank you, this is fascinating. Are you able to adjust the universe of polls that you're querying? Or like, is that something you would work into the query itself, if you want to exclude certain polls or certain types of polls or sample sizes, or?

00:31:48 | Jon Cohen

Yeah, so you can do that through here, if you want to say only these kinds of polls, only polls that are of this methodology. All of that is stored, basically kind of one of the, you know, Mark described a system where you can just go get all the most of this from the internet. The difference between the internet and this is it's basically, you know, we take all the relevant information and then we're chunking it. And so it's basically both how you store the data and the system of retrieval that kind of is part of the system. But yes, we could, do you have something you'd like to ask?

00:32:20 | Mark Murray

We have time for one more question before we get to the next part. Barbara from Marist. We have microphones coming your way.

00:32:27 | Audience member (Barbara, Marist; surname unconfirmed)

Hi, good, thank you, thank you for organizing this and thanks Jon, for showing us your latest and greatest. We appreciate being here. But what is the statistical analysis going on, kind of I'm assuming behind the scenes and what other types of analysis might be on your roadmap down the road?

00:32:48 | Jon Cohen

Yeah, that's a great question. So what you see here, there's no statistical analysis. Right, I mean, I talked about the advanced averages, there's some that happens there. Kind of one of the reasons why, one could sleep overnight publishing these polls is these are just the numbers that were published. Like and so it's like not asking ChatGPT in this case to calculate anything, right? So that's what you see here. Now in the case of the raw data that I showed you, there is a calculation behind it. And it makes really smart decisions. I think it's a great time to be a pollster. I don't know if it's a great time to be a data scientist, because these tools are so good.

00:33:24 | Jon Cohen

And you can also then tell them different things that you wanna do. So kind of like to do that over time, or even the charts that I showed about weighting, those were all logarithmic scales that it shows. Probably smarter than the ones I would have chose, or at least as good. And so kind of, again, part of my plea to experiment is you'll start to understand where you should trust it, where you should kind of step up, and where you should nudge it.

00:33:52 | Mark Murray

Thank you, Jon, that was amazing. And if you have other questions, we're gonna, at the very end, be able to answer it, but I'm now gonna turn it over to Michelle Jaconi for the next part of today's forum.

00:34:10 | Michelle Jaconi

Another pop quiz that's coming while Mark gets our panel ready. And this is about how many countries in the world, let’s talk about global reach of ChatGPT. And how many countries in the world have one third of the adult population using ChatGPT? So I want you to think about that. Where are there at least one in three adults as a weekly active user? You all got your number in your head. I'm not going to make you say it out loud, but I really want you to think, just guess a number. And Mark?

00:34:49 | Mark Murray

The answer is 77. The United States is one of them. All right, for the next part of our discussion that we have in the final part, we're gonna have a fireside chat with Brent Buchanan, the founder and CEO of Cygnal, Republican polling firm, and Ilana Ron-Levey who is the managing director at Gallup. Thank you so much for joining us, Ilana, Brent. Thank you. I want to kind of start off after Jon's presentation on how he uses AI as a pollster. Ilana I kind of want to ask you about how at Gallup for you, how AI has kind of changed a historic organization going back to the 1930s. It's been doing numbers for polling for a very very long time.

00:35:43 | Mark Murray

How are you guys using AI at Gallup?

00:35:48 | Ilana Ron-Levey

Well first of all thank you so much for inviting me. It's an incredible gathering and it's a little surreal for me. I've been at Gallup for 12 years. My office when I started was actually on this floor and it had the reputation of being like the most boring, the stodgiest floor at Gallup. So it's really incredible and surreal to see it completely transformed. That being said, like so many organizations AI is really transforming many of the ways that we look at our industry our workflows our client experience. On the polling side though I think there's two innovations I want to share and we're quite excited about it.

00:36:26 | Ilana Ron-Levey

In February of this year we published our first analysis of the large-scale experimentation that we're doing with AI phone calling. And what's interesting is I got a lot of questions. You know, does AI phone look more like RDD? Does AI phone look more like self-administered web panels? I think what we're really seeing is that AI phone calling is like a new mode. It has elements of, you know, the biases that can be introduced and mitigated through RDD phone. It has some elements of self-administered surveys where people are sort of needing to keep track of information.

00:37:16 | Ilana Ron-Levey

So it's really interesting to me that in, you know, such an established industry, like polling, we have new true modes with true mode effects being introduced. That being said, AI phone calling is promising on many levels. We see a potential for scale. We see a potential for very, very rapid deployment. Gallup does annual surveying in over 140 countries. So one of the pilots that we did, for example, was in Venezuela, you know, the day after the American operation. We were able to immediately gauge Venezuelans' reactions, which is exciting.

00:37:58 | Ilana Ron-Levey

So, you know, I think our view is you keep experimenting, you keep testing, you be radically transparent in what you're finding, and you go back to the basics of, you know, the mode needs to fit the research objectives. You lead with the research objectives, not with, you know, the fun and interesting mode of the day. We can get into this a little bit later, but we're also doing very serious testing about AI agents and synthetic data. And, you know, similarly, I think that keeping in mind those fundamentals, what is the sample composition in which the synthetic data is predicated?

00:38:40 | Ilana Ron-Levey

What questions is this appropriate to use for and not appropriate is something that will kind of lead us all in good stead to use AI responsibly.

00:38:54 | Mark Murray

Thank you for that. Brent, you guys just ended up coming out from the field with a new survey with a lot of data when it comes to AI sentiment. Maybe Lucas can actually stick it into our AI sentiment tracker, but what are some of your top takeaways from the poll that you just ended up having?

00:39:14 | Brent Buchanan

Yeah, it's rough as I think Lucas highlighted, and it's something that we really started to see about two years ago come up naturally in focus groups that weren't even about AI and weren't about data centers, and where people were getting their information and the things that they were saying that they saw about AI and about data centers was just challenging because there are so many inputs. It's not like this is a clean thing of like do you like this new technology or not?

00:39:44 | Brent Buchanan

So, one of the things that we did in this particular survey and we'll put a QR code up at the end where you can grab the whole deck with the analysis and everything in it, was really start with open ends and then do it at a large enough scale that we could code those and basically group people based on what are the emotional sentiments behind why people fall into these categories. And we ended up with five groups, the first being FedUp, about a third of voters.

00:40:12 | Brent Buchanan

Doomers at almost 20 percent, Boosters at an abysmal 16 percent, Guardrail Seekers at 10 percent, and then everybody else a little less than a quarter that really didn't have much of an opinion at all. And so, that framed a lot of how we then looked at the rest of the survey. And it's not a huge survey. It was, you know, we do a monthly political survey. This is midterm voters. It's not all GenPOPs. So, there are some aspects to it that you've got to keep in mind of what the sample was. It was really interesting to see that the public is – because we like to look at things through the emotional lens.

00:40:45 | Brent Buchanan

And I think a lot of the AI conversation has been a logical conversation. And so, we're bringing a little water pistol to a machine gun fight right now. And the negative emotions around AI, which is what a lot of the survey showed, you know, we can't come to them and say, well, if you only knew this, if you were smart enough to realize X, then maybe you would actually like AI. And my assumption was that going into the survey is that we were going to have a lot more folks who were anxious. And I was expecting a whole lot more fear as kind of the top level emotion, being driven by anxiety as a secondary emotion.

00:41:27 | Brent Buchanan

And it ended up being a whole lot more resent than it was fear. And so, that surprised me. And for those of us who are, one, using this, but, you know, we get hired by companies to help change the opinion too, not just measure it, and understanding how these people that even the boosters are a bit challenged in how they see the public opinion. And they're worried about the public harms and not the personal exposure. So, that was something else that surprised me a little bit. I expected the...when we said here are expecting five years AI's going to make this better or worse. We ran them through a grid on that.

00:42:04 | Brent Buchanan

And the ones that were more personal, I expected them to be at or higher than the collective societal concerns that people had. And it really came out that it was almost exclusively societal. Even people who fall into, you know, that fed up category still were way more concerned about the societal issues than they were on their own personal. Also, that the more people we can get using AI even just as an alternative to a search engine, like we're not, we don't have to have them building Codex flows to get them on board, but simple exposure to AI where they see some sort of nominal benefit to themselves is a strong predictor of hope. But it's not just showing them.

00:42:54 | Brent Buchanan

And that's the challenge. I asked a question. I think that maybe you had thrown out or Michelle, one of you too said add this in here. And it was, you know, if somebody sat down and showed you how to use this, would that change your opinion of AI? And that kind of bombed. People were, there were, there was not a large swath of individuals who were really interested in, you know, the genius bar approach to AI. So, requires rethinking, how do you get more people into using it for that? And then this, the fact that AI is moving too fast is not a partisan view.

00:43:30 | Brent Buchanan

It's three quarters of voters, and there's really no voter segment where it's under 60%, it might be right, a hair under among Trump Republicans, but even they, who are the most relatively supportive, if you divide out the partisan scale by, you know, tribes within each of the three. And that a lot of this is from within the backdrop of affordability. So a fight of which we have almost no say over and honestly, I mean, I'm a tech optimist too myself, like, I do believe it's going to drive down costs and we potentially see deflation for the first time, positive deflation for the first time in at least since I've been alive.

00:44:13 | Brent Buchanan

That is the backdrop in which all of this conversation is going on. It gets drug into the AI conversation too. And data centers, we've gotten there also.

00:44:21 | Mark Murray

I want to further explore the comment you made about a resentful public to this kind of technology. Ilana, you and I were talking about a week ago as we were doing some prep about just the overall general pessimism in this country. I see Gallup obviously has been doing right track wrong track tracking for decades. My friends over at Hart Research, I see Jeff Horwitt for going back for decades as well.

00:44:46 | Mark Murray

When you end up looking at right track, wrong track, the wrong track is close to at all time high, if not at all time high, this is a very deeply pessimistic public with a new technology, and Ilana, I was wondering like, do you kind of see any kind of how that fits into what Brent was talking about?

00:45:05 | Ilana Ron-Levey

I'm so glad you brought that up because I think kind of looking at public receptivity to AI divorce from the overall information, social cohesion, polarization, climate in the U.S. I think is a mistake. I think you're right. I mean, majority of Americans have been becoming increasingly pessimistic about whether or not the country's on the right track. We ask every month, an open-ended question, I hope some of you follow it, I don't have a QR code sadly, but about the most important problem facing Americans. We do a lot of analysis about how this sometimes can very powerfully portend electoral choices.

00:45:45 | Ilana Ron-Levey

Right now, we're seeing that government leadership's problems in leadership is the number one cited problem facing Americans with the economy second, and technology very, very low on the list. But topics that kind of take up the public consciousness sometimes are not viewed as most directly influencing people's lives but are still important to them. We also released data last week really showing that, yet again, Americans are not confident in traditional news media. Only 33% of Americans feel like newspapers, TV, print, really reflect the state of the world accurate and fairly. Social media does not have higher accuracy views either.

00:46:37 | Ilana Ron-Levey

So it's not a surprise that if you're pessimistic about the direction of the country, you think the leadership, and this is writ large. This is not about presidential approval. This is about the way in which the country is governed both congressionally and presidentially. Even other institutions like the Supreme Court, vast majority of Americans do not agree that they're doing a good job. So you're not confident that the leadership of this country performs its functions well. You don't trust traditional or new institutions as well as the accuracy or the fairness of the information that they cite.

00:47:16 | Ilana Ron-Levey

It would be shocking if all of a sudden, a new technology that contributes to public discourse that influences institutions, all of a sudden is going to be widely trusted and viewed as inherently very different.

00:47:32 | Mark Murray

Brent, with that pessimism, with the survey that you just ended up looking at, if you were to advise a Republican who was running for president in 2028 on, hey, how should I tack when it comes to AI, what would your advice be?

00:47:48 | Brent Buchanan

It would not be the techno-optimist approach, even though that can be personally what they believe. It has to involve guardrails of how are you protecting people in a responsible way from this technology and from the people behind this technology, and I think that's something we also can't ignore, which is that the big tech moniker is not just that it's a corporation, which also has no trust. Corporations have no trust at all. But it is run by people who are brand names in and of themselves that also carry into conversations how people feel about those individuals, whether it's Elon Musk or Sam Altman or Dario or anybody else.

00:48:32 | Brent Buchanan

And so the thing is like what you say, and I'm not advising that candidates should lie when they're up there, but you have to figure out what do you focus on or what do you not focus on to get the end goal so that you can be the person who's actually in charge of doing the regulation and rules writing. And that is the really challenging needle to thread at this current point, which is where is that balance? Because it seems like on data centers and AI kind of together as an issue that you can't divorce from each other, you just, you have a lot of legislators, state legislators specifically.

00:49:12 | Brent Buchanan

And I think that's where as much or more risk lies even in the federal, because if you end up with a hung government after this election, like that's probably the best thing that could happen to AI is that nothing actually happens in this country on anything, let alone AI, but it doesn't change the fact that 50 states are gonna be rushing on this, and the red states, blue states, the purple states, everybody in between, all you have to do is go back and look at how many bills were filed three years ago on AI and data centers and how many bills were filed now.

00:49:43 | Brent Buchanan

And then the other piece that goes on into all of this is that one of the main emotions that we picked up on in the survey is violation, like they feel violated by the technology. And that's a whole different aspect. And so now you get privacy laws into this whole mix too, and not just what do we do with chatbots and what do we do with the data centers powering the chatbots.

00:50:03 | Mark Murray

Brent, one quick follow-up. One place where AI kind of breaks even in your survey is on the issue of healthcare. And applications for healthcare. Explain a little bit about what you found there.

00:50:13 | Brent Buchanan

Yeah, so the grid was 11 areas of life. Healthcare's one of them. Healthcare's the only one that wasn't negative, but it was because it was 35% better, 35% worse. So it is a relative better, not an actual net better. And we didn't dig into it, but one of my theories, because we do a lot of healthcare work too, is that that's an area where people feel very trustful of certain parts of healthcare like their doctor and to an extent their hospital, but then when you get into vendors that serve both of those is where kind of healthcare falls apart from a trust perspective.

00:50:49 | Brent Buchanan

That because there is some trust in that institution, because it's such a broad, it's a fifth of our economy. Such a broad ask that that's why there was some positivity there because they see areas they like and they see areas they don't like within it compared to other items that we put within the grid.

00:51:10 | Mark Murray

I have a couple more questions before I turn over to the audience, but Ilana I wanted to ask you where you think three to five years from now survey research goes, particularly with AI and how you might be foreseeing some of that.

00:51:23 | Ilana Ron-Levey

Yeah, it's so interesting that the regulatory environment is going to be a huge component I think of the ability of AI to truly transform the collection side. I think there's no doubt, and I'm sure everyone in this room is experimenting to some degree and I think as an industry we need to share more about it, but there's no doubt that AI is affecting some part of the pre-data collection and the post-data collection process from automating certain parts of sampling to, I mean, ChatGPT is incredible if you're brainstorming about new survey questions and you wanna quickly see what are all the validated survey items that have been asked about X and save some time there and how you use lit reviews.

00:52:16 | Ilana Ron-Levey

And we're doing a lot of experimentation of using AI and some of the qualitative components for cognitive interviewing. So no question that pre and post, it will speed up data collection, it will raise the expectation of quicker out of the field quicker to accurate data sets. But on the collection side, the regulatory environment is gonna play a major role because how consent will change, will state have major issues with AI enabled phone calling for all sorts of different reasons. I think that is going to play a part. The other thing, there's a new AAPOR set of very important guidance, which I'm sure many of you are following about AI and transparency in survey research.

00:53:11 | Ilana Ron-Levey

And I think it just takes a few bad actors to not disclose when AI was used or try to pass off synthetic data as human that can really erode trust. I think responsible actors in the survey research industry understand that this whole industry is predicated on complete transparency about methods. But I think it will just take a few times that the public trust components, and of course, surveys and polling have been challenged in recent years. We don't need to talk about all the many reasons why, while leaders rely on polling, I think more than ever before. So it's a sort of dual side.

00:53:58 | Mark Murray

Ilana, you have this a perfect transition and my final question for both of you and kind of where do you see the fundamentals of the polling industry? I mean, obviously there are so many people that say, Oh, can't believe the polls, can't trust the polls. I wanna, for the professionals here and professionals in the audience, is what your quick take on the standing of the industry right now?

00:54:18 | Brent Buchanan

I wanna tack back on this prior question first. And I think we've got to say the quiet part out loud in that the panel quality has degraded so much in the last year or two. And when everybody has dots, how much faster is it gonna degrade when people are looking, they can make 10 bucks in Starbucks gift cards a day. Like you can fund your coffee habit by fraudulent panel access, you know, answering surveys and we're going to have, you know, every non-binary Hispanic 22 year old woman in Miami on earth is going to be there because there's like the highest paid individual within a survey panel.

00:55:00 | Brent Buchanan

And that's what scares me the most, honestly, about survey research is that we have these amazing tools that are helping us once we have responses. And what can we do with the responses? But what if all the responses are crap? And for those of us who do political research, we work off the voter file. So I've got 220 million records to work with, but what if I can't contact them?

00:55:22 | Brent Buchanan

What if I [unclear] strip from my files, things like race and other things that we even collected first person that's not even modeled attributes, like the sensitive data that these states, that’s what worries me the most about our industry is the ability to get good quality responses, either by panel or by direct RDD or text to web or whatever, the knocking on doors.

00:55:48 | Brent Buchanan

So not to be a doomer on this, but those are that's my biggest concern in what we do as a business, because I haven't seen nearly as much impact on AI there, because we still have to talk to a human at the end of the day, we've got to get a human to respond and finish the survey thoughtfully.

00:56:06 | Mark Murray

So Ilana?

00:56:08 | Ilana Ron-Levey

Yeah, I mean, you know, in 1935, when George Gallup founded Gallup, it was a pretty revolutionary concept that leaders should be held accountable to public opinion, and that the notion of representative samples to inform leaders about how the public really feels about issues really was a novel concept. We sort of take that for granted, and I think so many of us try to live out those values every day. And I'm also reminded just in recent years, how radically certain policy decisions have evolved, if it's marijuana legalization, if it's certain elements related to foreign policy. I mean, you see a reliance from leaders to understand public sentiment, and you see public sentiment rapidly evolving on a whole host of issues.

00:57:03 | Ilana Ron-Levey

And I think that that need to understand how the public feels won't go away. It may only increase in importance. I do think though, and I completely agree with what you said, that there's many sort of challenges to the validity of polling results. It can be from the sample composition, it could be transparency, it could be fraud issues, it could be not correctly using the right sample source depending on the topic, it could be overstating and overgeneralizing, I mean, you name it.

00:57:37 | Ilana Ron-Levey

But I want to believe that if the public is united, that it's important sort of enforcing and the public wanting to understand that the data that they're consuming are high quality, they do follow a set of standards, and that they understand where AI falls into it. AI is radically powerful for survey research, but the public just needs to understand where it would fall into the process, how is it used, how is it validated, and I think we ask this of everything. So maybe part of the equation is a little bit more public literacy in the fundamentals of polling so that people can evaluate for themselves.

00:58:22 | Ilana Ron-Levey

And, you know, when I was looking at Jon's demo and, like, those great questions that you can see that if, you know, an AI user just can't distinguish between what makes for a high quality or low quality poll, it could add to misinformation.

00:58:43 | Brent Buchanan

Yeah, that's what I was to answer the question that was actually asked. We have to inform the media that those of us here in the room need to make the media more literate on reading polls. And there are outlets that have people like you two guys who your job was to do this. You're the exception, not the rule. And polls are like catnip to the media. And they just jump on it and report it. And most of them probably never even looked at the demos. Like they looked at the ballot and wrote a story. And they didn't look at the name IDs. They didn't look at info flows.

00:59:15 | Brent Buchanan

They definitely didn't look at the construct of is this correct on partisanship, is this the retro ballot, like a believable margin. And so that's one of the things that we've been talking about as a firm is like, how do we help the media be better at this? Because to your point, they're the ones we've got to get to be better consumers of surveys that will result in the increased public trust in surveys, because it means some surveys won't get reported on, which is probably the right thing to happen.

00:59:44 | Mark Murray

Thank you for that. Any questions from the audience?

00:59:56 | Audience member (name unconfirmed)

Ilana, you mentioned Gallup is using A.I. to conduct interviews. Can you talk a little bit more about that, is that like A.I. interviewers replacing human interviewers to talk to humans? Like, how are you dealing with the regulatory issues around that?

01:00:08 | Ilana Ron-Levey

Great. We have a public methodology blog about our experimentation to date on news.gallup.com. If you search under methodology blog, you should see it. But where we are right now is we are partnering with an industry leading A.I. voice, sort of the equivalent of an A.I. call center, and basically looking at how will A.I. voice compare to human random digit dialing interviewers. And that's where we are right now. You know, no kind of commitment to replace anything. Really just trying to evaluate how do A.I. phone calls compare to human RDD and how does it compare to probability-based self-administered panel surveys. But what's been exciting is we have over 500,000 surveys conducted to date on A.I. voice across four different continents.

01:01:12 | Ilana Ron-Levey

So we're really, you know, obviously getting a sense of what are the country context where this performs well, what are the types of questions. And we're going item by item, as you can imagine. You know, we're running a survey, human RDD, self-administered panel, A.I. voice. And as I alluded to, I think, you know, the early indications are it performs differently. There are elements where it performs similarly to human RDD, elements where it performs similarly to self-administered probability-based panel surveys. That's what we're really trying to understand. For what kind of questions does it perform one way? For other kinds of questions, does it perform another way?

01:01:57 | Ilana Ron-Levey

On a positive note, one thing that we're seeing though, we survey our Gallup panel members, this is, you know, one of the country's largest probability-based panels, and we survey them about their experience because we want to understand. And there are, you know, we're hearing pretty positive things like finding the technology works well, kind of liking the fact that maybe some people feel like they can be more open and honest if they know that there isn't a human on the other end, because we all know for sensitive questions, there's some differences between self-administered and phone. And we're committed to really, you know, showing the world the results of these experiments.

01:02:40 | Ilana Ron-Levey

I'm sure at AAPOR next year, we'll be presenting a lot about what we found. We did this year and we'll have more to share, but, you know, the rapid deployment is obviously something really, really unique. So I think if I would summarize, I would say I think what we're trying to understand are what are the specific types of research questions that can accurately be answered by AI phone.

01:03:08 | Ilana Ron-Levey

I think that's where we are, but to your point, the regulatory issues will become increasingly important, because as you can imagine, we could see that it works well in country X and then in six months, the regulatory environment would change the consent process so much that what we're also trying to understand statistically is are the people who opt in, who are willing to consent to an AI phone interview, are they systematically different than those that don't? That's a really fundamental question that I think we'll all need to work together to try to answer.

01:03:49 | Brent Buchanan

We ran a test last year where we worked with a partner and built out an AI interviewer and it could ingest out or could take out our Qualtrics file, sorry, no offense, and it could actually administer the survey, follow all the skips, follow all of the randomization and flips and it did all that really well. Then we did several statewide surveys where we ran our regular phone calls and then we ran this. Our lawyer said you have to have a human initiate the phone call, so now we're manually dialing. You have to give them notice and then ask permission to transfer.

01:04:29 | Brent Buchanan

This is U.S. obviously and it was Matt, I don't remember, at least twice as expensive if not three times as expensive to do it and that's not even including the AI cost. That was simply just the phone cost between the two and so we shelved it because I'm not looking for ways to triple my cost.

01:04:48 | Mark Murray

I think we have time for one more question from the audience. Anyone.

01:04:58 | Michelle Jaconi (speaker uncertain)

How does the response rate change when you talk to people, you have to talk, you're talking to a human and now you have to talk to an AI? What was your—how many—what percentage of people [unclear]?

01:05:11 | Brent Buchanan

Was it at least two thirds of folks didn't consent? And I think that's where the math just worked out to where it tripled the cost because two thirds, and but what we did find is there was not a large difference between who actually took the survey. So they weren't like significantly more college educated or answered this question way differently than the people who were being just the full human interview, so that was positive, assuming you wanted to pay a bunch more.

01:05:42 | Mark Murray

Brent, Ilana, thank you very, very much. I think we also want to put up the QR code for Brent's Cygnal analysis, so definitely take a picture. We want to have Brent's great analysis there, so thank you guys very, very much.

+ Read More

Comments (0)

Popular

Watch More

AI Ethics in Action: UC Berkeley’s Data Science for Social Justice Workshop

Posted Aug 30, 2024 | Views 10.2K

# Social Science

# Higher Education

# Socially Beneficial Use Cases

Event Replay: Sam Altman on Building the Future of AI

Posted Apr 06, 2026 | Views 10.9K

# OpenAI Leadership

# AI Governance

# AI Safety

# Economic Opportunity

Event Replay: Minus AI, Plus AI, Times AI — A Vision for an AI Pedagogy

Posted Nov 14, 2025 | Views 1.5K

# AI Education

# AI Pedagogy

# AI Adoption