Event Replay: Inside OpenAI's Investigation into Foreign Influence on U.S. AI Debates

[00:01:00] Welcome everybody, and thank you for joining today's open AI forum. I'm Chris Nicholson. I'm on the global affairs staff at OpenAI and we are excited to be joined by two of my colleagues who are on the intelligence and investigations team. They're called I2. Let's talk about Albert Zhang and Ben Nimmo. Today we're going to talk about I2's work at OpenAI and their latest reports on PRC-link to influence operations. That's PRC as in the People's Republic of China, it's mainland China linked to influence operations that are targeting AI debates in the US. They're trying to influence American opinion about AI. Now I'd like to open this up to my colleagues. They're here. Let's start with Ben. Ben, can you share more about the I2 team's work at OpenAI? What does your team do and what role does your team play?

[00:01:58] Ben Nimmo: Absolutely, and thank you, Chris, for the introduction. It's great to be back here again. So I2 is one of the teams at OpenAI that work on keeping AI safe. There's a whole range of teams we have, all the way up from, like, the model training, the safety teams whose job is to say, hey, some things are so dangerous that you just shouldn't let the model do them at all, and so they train the model to refuse. But there are some areas where there's a gray area. Maybe you're asking the model to generate a tweet or generate a picture, and like maybe you're an innocuous person, maybe you're a threat actor, and so we are on it with the next line of defense. Our job is to go and try and detect people who are trying to abuse AI, trying to use chat GPT or use our API in some way for harmful purposes. We have specialists who deal with child safety, for example, we've got experts who work on scams, terrorism, violent activities. My main specialization is covert influence operations, so foreign interference, election interference, people who are trying to use AI to pretend to be something they're not to have some kind of political or social impact. We really think of ourselves as the team whose job is to catch the stuff that the other systems have not caught and then flag it and say, okay, how are we going to respond to this? How do we make our entire product, our entire world safer by doing the work we do? So it's hard to draw bright lines when people are creating AI, so they need a lot of human judgment applied to see when something falls on the side of abuse versus legitimate use.

[00:03:37] Ben Nimmo: Something we see, my specialization is influence operations of the classic kind. You might see it long before the days of AI, it was a Russian troll farm where you have a couple of hundred Russians sitting in a room in St. Petersburg, all running social media accounts pretending to be Americans. Most of what they're posting on social media and most of what they were posting long before AI wasn't false claims, it wasn't particularly hate speech. It was much more, "Hey, I think this candidate is better than that one." It's opinion stuff, it's nuanced. It's not really, you know, the harm doesn't come from the fact that somebody says, "Hey, I prefer this candidate." The danger is in the fact that it's a Russian who's running a fake social media account pretending to be an American and they're posting them online. They're trying to sort of manufacture this fake image of consensus or the weight of opinion in one area. So if you take that sort of understanding and you map it onto the AI world, you, as Chris, could ask a chat, "Hey, I support this candidate and I don't like that candidate. Write me a tweet about it," because you can't think of one or you haven't got time to think of one on your own. And then you post it from your own account or Instagram or threads or whatever, and that's totally fine, right? You're just using AI as a creative tool. If somebody who's associated with the Chinese law enforcement does almost exactly the same thing and they generate a tweet in English which says, "I support this candidate, but not that one," and then it's posted by a fake account on...

[00:04:58] Speaker 1: and then it's posted by a fake account on social media, they're trying to interfere in the election.

[00:05:03] Speaker 2: But what the model actually sees is something very similar. And it's not the kind of thing where you're going to say, well, we're not going to allow anyone to generate any tweets on any political topic. That would be overkill. So we have to have a different way of sorting out what is the authentic activity from what is the sort of the coordinated deception that my team will look for. And that's our job. We've got tools built, we've got processes built, we've got a lot of AI working to help us sort out what is the small minority of threat actors who are floating around in that sea of innocuous activity.

[00:05:36] Speaker 1: Yeah. And we have this asymmetric situation where America is more at risk to such interference because we have an open society that values open debate, but that also lets some of these operations creep in. So what is it, the coordinated misuse of AI, like more than just a guy in a room like me, but it sounds like a troll farm. There's lots of people. What is the coordinated misuse of AI look like? What kinds of behaviors manifest?

[00:06:04] Speaker 2: It depends from operation to operation and for 2024 we've actually published our findings on some of the influence ops and scams and harassment campaigns and cyber campaigns that we've disrupted. So we're trying to put information out there in the public space. You can see from there that there's a variety of ways that operations will work. Some of them really do feel like you have maybe a dozen people in the room and each of them has got a chat GPT account and they're each generating content and normally they're generating content which is posted by a bunch of different social media accounts. Sometimes you get an operation where it looks like it's only one chat GPT account or it's only one API account which is generating content for a whole load of different social media accounts at once.

[00:06:50] Speaker 1: We had a particular striping operation back in October we reported in October 2024. Somebody was using the API and effectively what they were telling the model is here's your personality read this tweet and generate reply now here's a different personality to the same thing and they were running what looked like hundreds of different social media accounts off platform.

[00:07:10] Speaker 2: So there you have one single access point to the to the AI but it's being used to generate content for a whole load of different accounts on social media and so it varies from this kind of very manual. Sometimes you get what looks like one user using chat GPT to run maybe half a dozen accounts on social media so it can be from the very small to the more scaled. One of the interesting things that we've seen is that you know I saw your eyebrows go up when I was talking about the big API operation because instinctively for a lot of people that sounds like wow that's that's scary because it's big and yeah it's big but also because it's big it creates a really big signal that you can detect and if it's relying on AI and you take it down then you've broken everything it's doing all at the same time.

[00:08:00] Speaker 1: So this operation that we took down back in 2024 when we banned it we knocked the entire operation offline for about 10 weeks because they were you know we've stopped their access to our AI and they're having to work out what to do next. So you get different models of activity different types of activity within this kind of threat space. Our job is to understand what does each one look like how do we detect it how do we not just ban it from our models but disrupt it so that we can actually have a knock-on effect we can expose it we can show other people here's how this works because one of the things which has been true for influence operations for a long time if they're trying to stay hidden then they don't want you to expose them and if they get exposed then it makes it much harder for them. And so part of the work we're doing is this sort of expose.

[00:08:48] Speaker 2: For sure.

[00:08:51] Speaker 1: Yeah the last thing I get it that makes total sense. So maybe you can help us connect the dots. Why would the employees of a foreign government that is an adversary of America in some domains and theatres, why would they want to run an influence operation in America? What are they trying to do?

[00:09:09] Speaker 2: I mean there have been lots of different influence operations over the years and to be fair they don't just target America. You know in Arctic dance we've had influence ops targeting many different countries some of them foreign some of them actually domestic as well. You know we've taken down domestic operations in countries like Rwanda and the Philippines. But in general again you know since before the emergence of chatGPT and since really AI broke onto the scene influence operations seem to be a way of trying to exert political influence in a country without saying that it's you. And this was this was the great opportunity that ThreatFactors saw in social media. You know if you want it traditionally for hundreds of years countries have tried to influence each other and you try and get your own agents on the ground and you try and sort of influence people there you talk to them. But it's quite hard if you come from a country on the other side of the world to like

[00:09:56] Speaker 1: Come from a country on the other side of the world to like to blend in with local population and suddenly with this thing called social media you can steal somebody else's profile picture, you can give yourself a false name, and you might think that you can blend in much more easily in that population and start influencing people while pretending to be one of them. Now the reality has proven to be over the years it's a lot harder to pretend to be a local than people think and you know particularly influence operations from Russia and China are legendary for how bad they've been at English and they would try to run I mean I remember one particular Chinese operation from sort of 2018-2019 was running a lot of fake accounts on YouTube that were posing as Americans and one of the videos they ran had the title I'm paraphrasing slightly this is as close as I can get the water of American democracy can no longer sustain the boat of presidential rule. When you look at it you're thinking you're pretending to be an American writing about this that doesn't seem very idiomatic and so that there was the idea that it was easy to do it for interference with social media then you have the realization that actually it's pretty hard to do it and not get caught and so when AI came along I think for the threat actually with this moment of thinking oh well maybe if what we're doing is making language errors one thing that large language models are pretty good at is not making language errors so maybe if we switch to using AI that'll give us an advantage and so what we've seen is some of the operations which have been running for years in different forms started trying to plug chatty BTN or are they our models not just us but started trying to plug them into different parts of their workflow to see if they could reduce that error rate a bit to see if they could get any better at what they were doing what they hadn't thought through was that if you're using another platform like us then you're showing up on the radar of another investigative team like us so there are more people who can find you and one of the things that I'm actually proudest of as an investigator is if you look over all the operations we've exposed quite a few of them are ones that hadn't been reported before whether it's new threat actors or ones who had just you know started out and started to try AI we exposed them and so actually what should have been an advantage for them turns into a disadvantage and what we're always trying to do is think how do we use our AI to get better at catching this kind of bad activity and how do we make the inconvenience of getting caught higher than the convenience of using the model in the first place and so a lot of what we think about Albert Nye is how do you disrupt a threat most effectively.

[00:12:31] Speaker 2: Yeah, there was the old joke in the early days of the internet that it was a single panel joke at Golden Retrievers talking to a pool thing in front of a laptop saying on the internet nobody knows you're a duck, right, and so it sounds like on the internet maybe nobody knows your intelligence officer running a disinformation campaign except when you're pretending to be ten people at once targeting data centers, in which case we know.

[00:12:49] Speaker 1: If you're using our model or if you get caught by one of the other platforms.

[00:13:01] Speaker 2: Right. Let's see, so Albert, China's not the only actor using these tactics, do you see parallels between, say, them, Russia and Iran and other nations that might want to disrupt things or confuse us in America?

[00:13:17] Speaker 3: I think with the, like, threat reporting that we've done in the last couple years, we've exposed dozens of different networks from China, from Russia, from Iran, and various other sort of countries around the world. And maybe to sort of clarify, the way we kind of investigate these sort of threats or misuses of AI is based on behavior. You know, what we're looking for is COVID influence operations. You know, these attempts to sort of, I guess, manipulate public opinion or interfere in political processes while sort of hiding kind of who you are, really. So we do see that kind of type of behavior consistent across all those different types of countries. I think what's different for each different actor is that you've got to remind yourself that you've got to put your hat on, as if you are a Russian intelligence officer or a Chinese public security officer. You have different sort of goals or different incentives. You know, if you're the PRC, you're thinking about trying to control the narratives around how the Chinese Communist Party is perceived. Maybe if you're Russia, you know, you're trying to soak a school in the US. Maybe you're trying to limit, sort of, political support for the Ukrainians in kind of your war in Ukraine as well. So we see those tactics being employed in different sorts of situations. But I think it's good to remember that the goals for each different actor is very different, and that's why it's worth kind of calling out those actors in those situations.

[00:14:47] Speaker 2: It sounds like American public opinion is a front in the war, in the Russia-Ukraine war or in, kind of, the tech race. Like American public opinion is actually going to influence how America does in a tech race vis-a-vis the PRC, is that right?

[00:14:50] Speaker 3: Yeah, absolutely. So, you know, as the Chinese Communist Party, kind of, specified in that later.

[00:14:54] Speaker 1: ...kind of specified in that latest five-year plan published in October. They outlined AI being kind of a really important strategic area for competition and for security. And so, you know, when that came out in October last year, we weren't surprised that we saw kind of influence operations originating from the PRC trying to interfere in kind of the public opinion debates in the US about AI. Because, you know, I think one intention is to sort of, one, sow discord and sort of kind of distrust of US tech companies and democratic institutions. That slows down the US and its ability to build out its AI infrastructure, and that gives China a strategic advantage.

[00:15:36] Speaker 2: Isn't the Communist Party in China actually doing this to its own population? Are we actually getting a taste of what it's like to live in mainland China and suddenly feel confused and oppose things that might deliver life-saving cures for us? All of a sudden, we think a lot of things, or we think that our compatriots oppose things that we know actually we're using every day and that we see as good.

[00:15:59] Speaker 1: No, I think this is a really good point, because, you know, I've been looking at these PRC-linked influence operations for multiple years, and when we tracked some of these original networks, this is pre-kind of the AI era, there was a network called the Spamfludge Network that the FBI actually had attributed to Chinese law enforcement back in 2022, 2023, I believe. And when you look at that particular network, Ben, actually my colleague, actually named that actor Spamfludge, so you can blame him for how difficult it is to say and spell in reports. But actually, when you looked into that network, before it sort of, I guess, tried to target American debates and tried to interfere in democratic societies, a lot of the activity on social media was actually in Chinese language, trying to target Chinese citizens who were crossing the right firewall into Twitter to get around some of the censorship that we're facing on Chinese social media and having their own independent discussions on Western social media. And the Chinese government is basically running these influence operations on Twitter and other Facebook, other social media platforms to try and extend their surveillance control of their domestic platforms overseas as well.

[00:17:09] Speaker 2: And I think open AI really cares about this battle between democratic AI versus authoritarian AI. Democratic AI, for us, is widely accessible AI to as many people as possible to enable them to be free and pursue their own goals. Whereas when we see how the PRC's deploying AI, it's very much to centralize and concentrate their control over all spaces. And that, for me, I think is really interesting, because it pushes this authoritarianism into totalitarianism, where the Chinese Communist Party wants to control what you think, what you do, and not leave any kind of personal space for you to have free or independent goals to pursue.

[00:17:50] Speaker 1: Yeah. Scary. Okay, so there's an anti-data-center campaign, that's the data-center bandwagon campaign, I think you called it. And there's another cluster, right? So what are the two gists, what are the two clusters and what are they doing exactly?

[00:18:06] Speaker 2: Yeah, so a bit over two weeks ago, we published a new threat report that disclosed two clusters of Chattarooty accounts that we had disrupted. Both clusters originated from the PRC and they're both interested, I guess, or trying to interfere in very legitimate US debates around AI policy, around tariffs, as well as broader US conversations around how US democracy is going, really. So the first particular network, we named it the data center bandwagon because this cluster of accounts was generating social media tweets, as well as images that were claiming that US data center build-outs were increasing electricity prices and this was being passed off to everyday Americans. And then what we found was that these particular social media content was then being posted on social media platforms by accounts that looked like Americans. So the way we kind of, I guess, uncovered that this probably was not legitimate and it was foreign in nature was because one, the actor on Chatuchak Tea was interacting with us in Chinese language. They were active during Beijing business hours, you know, when most Americans are asleep. We had indications that this network was probably linked to a Chinese private technology company. And based on some of the reporting that they were using our models to edit and generate in draft, they suggested that this particular company was actually probably working for provincial level Chinese government entities. So it's sort of like the state level rather than kind of like a federal level equivalent based in sort of the PRC. And maybe just to kind of emphasize and clarify, you know, we're not trying to claim here that all of the debates and maybe criticisms of the US data center build-outs originated from foreign actors.

[00:19:52] Speaker 1: ...from foreign actors. These concerns and conversations we were having long before this operation began, we believe so early this year. And I think the interesting part is that the distributor actor, based in the PRC, saw this as an opportunity to try and interfere and maybe further build distrust and discord in this particular debate. Some of the tweets that we looked at have many like three views or three likes. I suspect that one view was myself, another view was Ben when he was just checking my work, and then the third view was probably the operator themselves in China just checking to make sure the post was actually uploaded. So we don't want to overemphasize the impact of this particular operation, but it was important for us to disclose this particular operation because it says a lot about the intent of PRC actors and maybe an indication of other operations that may be running across the U.S. We know the Chinese Communist Party is very top down directives. And so when one part of the system gets given a directive to run a particular operation, we're likely to see those narratives or those types of operations appear in other parts of the system too.

[00:20:56] Speaker 2: Yeah, yeah, and I appreciate the nuance you just introduced. It's not that there's a debate to be had about data centers in America and there are legitimate points of view on both sides. And the most important thing to do is get the facts out there and let people air their positions. But the worst thing that can happen is if as we try to do that, foreign influence operatives are pouring kerosene on the flames and sowing confusion. That is not a great democratic process. So let's get back to this. Ben, I think you were talking me through some of the points earlier. You're putting out these reports. You've identified 61 different misuse operations over the last two plus years. The reports go out. What type of impact do you see? What kind of reception is there like in the public, among policymakers? Where do you see this go?

[00:21:54] Ben: So a lot of the reason that we do these reports is because when we started them sort of very early 2024 was at a time where lots of people were speculating about how AI might be used for various types of threat, you know cyber-influence operations, scams, and so on. But there was no real information out there on how it was actually being used. And you know what we wanted to do was to say okay all the conversation around how it might be used needs to happen because that helps you prepare your defenses, but right now what we're seeing is this. And we were trying to sort of fill in some of the blanks in people's knowledge around, you know, around misuse of AI in different fields. The reports have been, you know, whenever we have briefings with, you know, in D.C. or, you know, I'm in Brussels right now, they're welcomed because it shows what we're seeing and and by the fact of knowing what we are seeing you also know what we're not seeing in the sense that a lot of the operations we see, you know, to Albert's point, they're generating social media content but it's really not getting much traction.

[00:23:05] Ben: And it turns out, you know, it's a hard and lonely life for a fake account on social media because you're not just up against, you know, platform defenses, you're not just up against guys like Albert and me, you're up against all the teenagers out there who want to be influencers and are spending, like, far too many hours a day on their devices posting stuff. And it's really important to be able to kind of lay a baseline of here's how you would assess the impact of an influence operation, here's how we do assess the impact, here's what we actually see them getting and we'll draw a distinction between, you know, cases like data centre bandwagon where, as far as we can tell, they were operating on X and Facebook and nobody was really picking up on it. And you can compare that with some of the operations we've discussed previously where they were managing, like there was a Russian operation targeting the sub-Saharan Africa we exposed earlier this year, where they were running a fake persona as a PhD holder from the University of Bergen in Norway and they were planting articles in the mainstream media in sub-Saharan Africa under this guy's byline. And so, you know, you look at the difference between a tweet which has three views, two of which are Albert and me, and an article which has gone out in a big newspaper in South Africa and there's a huge difference in how many people that's going to reach. And we think it's really important to show here's how you can try to assess what the impact is. They're not all the same. AI operations are not all clones of one another. They have wildly outcomes. We even had an operation that we exposed again in February, again, a Russian operation doing sort of generic anti-Ukraine content. And the way this operation worked, part of what it would do is it would serve as a content farm. So one user would use ChatGB to generate, you know, a list of five or ten tweets and then different accounts on the internet, mainly on...

[00:24:50] Speaker 1: accounts on the internet, mainly on X, would post them. As there was one particular time when this operator asked the model for six tweets or seven on the Ukraine topic and we were able to identify all six of those tweets being posted on X within the space of like half a day or something. The tweet with the most views had 155,000 views. The tweet with the least views had 57 views. Not 57,000, just 57. And those two tweets are generated by the same model with the same prompt at the same time. Why did they have that difference? Because one of the accounts doing the tweeting had 800,000 followers and the other one had 800. So what we're trying to do with all this reporting is to describe here's what we're seeing. Here's the reality of what we're seeing from our little slice of the AI world. Here's the reality of what we're seeing. And sometimes it means that we can expose stuff that other people can then pick up on.

[00:25:47] Speaker 1: So in the end of 2024, we disrupted a scams network in Cambodia, one of these classic scam centers doing romance scamming. And they were using chat to generate flirty comments, which were then posted on social media, a bunch of different platforms, including Facebook. And we reported it at the time, and we shared information with Meta on this. Meta then investigated based on what we'd seen, and they did a public report crediting us for the initial lead and saying this was a newly set up scam center in Cambodia. And they'd done a big takedown far more than we'd sent them. And they also shared indicators back with us, which enabled us to go and find more scam centers. And so part of the exposure work is, it's disruptive to take down their chat TPT accounts, but if we can expose what they're doing, and if other platforms can then also take action, and then they can share stuff back, the rising tide raises all the boats. So it's all about trying to raise the defenses everywhere.

[00:26:56] Speaker 2: I'm learning so much. Today I learned you can have a banger in this information. You can really make a splash on X and it sounds like provincial government officials. So, it's like the state of Nebraska trying to disrupt the UK. It's very odd to me. There would be provincial operatives trying to come in here and mess with American public opinion. That's kind of funny. Let's see. We have some, it's 1143. I have some questions coming in from the community. We've got Daniel Green here and Daniel asks, I'd love to hear this from both of you. We'll go to Ben first. What role do AI tools like codex play in your investigative work? How are you scaling your investigation with our own tools?

[00:27:48] Speaker 3: I mean, one of the amazing things about working here is we get to use AI before anybody else does. We get to work on the best tools we have and it's really supercharged us in a couple of areas. One is detection. Back in the day when Albert and I were doing detection, you'd kind of have to know exactly what you're looking for and then go and look for that precise, be it a text string, be it an image, be it a meme because there was so much noise on the internet. If you did, oh, I'm looking for something which looks a bit sort of spam-muflagy, let's say you'd get like 99% false positives. AI has given this incredible ability to basically have a sort of tool which says, I'm looking for this kind of thing and here are the rough characteristics, go find it. And the agents will do the finding and then we have secondary sort of investigators that will run over the findings and say, well, that's a false positive, here are the true ones. And so it's made us just orders of magnitude better at detecting activity at the same time as it saves us time and it means we're not looking into half as many cases which are false positives because like we're only getting to see the things that are the highest confidence.

[00:29:03] Speaker 3: And the other thing, the AI is just wonderful for is like doing work on one thing while you're doing work on another. So codex particularly, you can set it up to say, okay, I need you to do an investigation analysis on this, go do it. It's probably going to take a couple of hours to do it. While codex is doing that, I'm doing another investigation. And so it's like each of us has like a couple of extra investigators in our holsters and you can just pull them out and get them running and then you get on to do your work. So it really is, it's a game changer already and I think we've only started.

[00:29:27] Speaker 2: Right. So it's bringing transparency to like the big mess of data that we all kind of confront at work and like it's finding needles in haystack and you have to be searching for some very interesting needles. Did I hear you use the word spamouflage?

[00:29:44] Speaker 3: Yes, so spamouflage is the contraction of the word spam and camouflage. This is the name of the Chinese...

[00:29:48] Speaker 1: ... and camouflage. This is the name of the Chinese operation which I, to my shame, named in 2019. I was the first researcher to identify it and at this time the operation was on Twitter, Facebook, and YouTube and it had this pattern where they were all accounts posting in Chinese and sort of trolling Chinese dissidents. They would do one political post and then they do about four or five completely spammy posts about, you know, here's a lovely Sichuan dinner or here's a beautiful mountain in the Midwest or something. It was pretty clear looking through it that the point of these spam posts was to camouflage the activity, so they weren't just doing politics all the time. So I nicknamed it spam-of-large and the name just stuck.

[00:30:31] Speaker 2: I love that, I love that word. Okay, now, how do you use Codex anymore?

[00:30:37] Speaker 1: Well, I'm using Codex at the moment, right? One of the great things about these are genetic tools that they can now operate proactively and kind of almost like almost independently of the investigator's control. So, you know, I only have 24 hours during the day and eight of those I'm hypothetically sleeping, I guess, really. As we're speaking, my Codex agent is going and triaging through our data, flagging kind of cases, flagging networks I think are high risk or match certain kind of taxonomies or profiles that we think are likely linked to threat actors who are running COVID influence operations or doing kind of other malicious uses of AI. The second thing which I think I want to call out on Codex and kind of our models more generally is that they are really now really powerful open source kind of investigators too. When this particular private technology company was running kind of this COVID influence operation against US data center debates, I just plugged in the name into kind of Codex. I said go and look at all the different Chinese language sources and help me generate a due diligence report about this particular entity in English because I want to share with my colleagues as well. I think this is really powerful as well to sort of like I guess protect the information environment more broadly, right? When you hear facts or hear claims, Codex lets you kind of do your due diligence, investigate the background of a particular entity in a way that maybe you had to outsource to an expert, to a consultancy group, or to other highly technical teams. I think our models are really good at navigating the internet to actually help you accumulate information for you to then assess and kind of decide whether you should believe a piece of this information or to maybe find the origins of where this information came from as well. I think those are really powerful tools that should hopefully encourage people to sort of, I guess, take the truth and maybe find other COVID operations.

[00:32:19] Speaker 2: Yeah, fascinating. So in the old days of machine learning, we used to like press train on the model before we went to sleep. And you're saying these days before you hop on a call with me, you're pressing kind of send on the agents to get them to keep doing work while you're away. Is that it?

[00:32:34] Speaker 1: Exactly. I mean, I have like a general guideline for my agents and agents would go in and kind of find stuff like that. So when I come into work and I sip my coffee, I've already got like five potential cases I should review and maybe dive deeper.

[00:32:49] Speaker 2: Yeah, amazing. Okay, we've got a question from Jason DeLuca. What signals help you distinguish these campaigns from normal political debate and which detection models can you describe publicly?

[00:33:01] Speaker 1: I think I'm very glad that you included the word publicly there because we have to be careful on how we describe exactly how we detect. But we look at as many behavioral indicators as possible, for example, what language are you prompting in, what time zone are you coming from, what kind of things you're focusing on, what else are you telling the model about yourself. There's a whole range of indicators that we will use and particularly, we'll get agents to do all that for us. What we want to happen is like the agents weed out all the false positives so we're only ever looking at the true positives because that's best for everyone. If you think about the pattern of, let's say, a Chinese influence operation, you have a bunch of people who are hands-on laptops somewhere in mainland China who are prompting in their native language. At the other end of the chain, you have a bunch of social media accounts that they're running which are posing as Americans posting in English about very specific political topics. This was not spam-of-large; that's kind of the only thing they were doing. That's a really different behavior pattern from a real person who has a political view and is saying, "Yeah, I need help to write a tweet because I can't quite think of the phrasing." One of the things we are always looking for, and one of the things we always insist on in investigations, is when you're looking at a suspect you have to bring in, like, what's the alternative competing hypothesis? How do you disprove the theory that this is just somebody in their own country commenting for their own reasons and it's entirely legit? A lot of our work is around making sure that we're doing that due diligence and ensuring we're not just finding stuff and thinking, "Oh, it's bad, take it down." We have processes that we go through; we have internal... [00:34:46] We have processes that we go through. We have internal approvals that we have to get. We have to present the case internally. Here's what I found. Here's why I think it is violating. Here's all my evidence. And we will check each other. And we will pull on all the threads we can just to make sure, okay, are we really sure about this? And sometimes you get an investigation. You're halfway through the case and you think, yeah, no, this is not something violating. I'm just, I'm stopping it right now. And I'm just like deleting that. And I'm going to go back to my detection and find something which is worth, more worth spending time on. So, yeah, it's a fair question. And a lot of what we do in sort of our internal processes really kick the tires on the vehicle to make sure that we are focusing on the true positives and we're not sucking up false positives by mistake.

[00:35:34] Yeah. I also emphasize that, like, you know, we're not really assessing the content based on the activities, right? We're really focused on the COVID nature of the fact that they're trying to hide that particular agency. So, for example, you know, this debate around US data centers, if Xi Jinping wants to come out and say he doesn't like the US because it's building data centers, he's welcome to say that, right? Like, we're a democracy, we have these open debates, people can kind of contribute to these discussions. But, you know, if Xi Jinping is creating a fake account on Twitter to then kind of try and pretend to be an American and kind of like spread that kind of narrative, then that's when things become problematic and that's the type of deception that we're kind of detecting for.

[00:36:12] Cool, okay, I've got a really interesting question here. It's from Alex Van Buskirk, a public ethics investigator at the city of Oakland. And Alex is asking, without opening eyes internal platform data, what kind of open source intelligence could be used to best distinguish organic political activity from coordinated deception? What would you look for?

[00:36:36] So, in terms of like foreign interference, you know, I've been studying the Chinese Communist Party for multiple years and when I try to look for foreign interference operations originating from China, you should always start with kind of the actor, right? Start with the Chinese Communist Party, understand how the Chinese Communist Party's United Front strategy kind of operates. So, if you're unfamiliar with that term, I think there's some great reports that you can look up about United Front, but the idea is really to try and, you know, build friends and, I guess, target enemies outside the Chinese Communist Party to sort of advance your interests.

[00:37:10] Just to intervene, I think the question is more general. Like, let's say there's disinformation campaigns that are coordinated, right? They could be originating within America, right? But how do you differentiate between organic activity and coordinated deception?

[00:37:27] I think the, I mean, my point about kind of the actor attribution is really, really important, right? Because if it's a foreign actor, you know, you wanna basically start there, see kind of what activities, are they running into kind of the US or into domestic kind of debates really, right? Likewise, if there are discussions in kind of like, you know, the domestic kind of debates, you just start with, okay, who is sort of contributing, who's sort of offering their views, maybe, you know, an understanding kind of why they're sort of offering their views. If people are engaged in good faith, you should be respectful and kind of consider their opinions but, you know, if there are financial interests, maybe there's state backing, maybe there's foreign kind of involvement. That's when things, you know, should be questioned. And, you know, the kind of the mitigation is cases, not necessarily sort of censor or shut down kind of these debates but actually just more transparency around kind of who people are, why they're saying the same things and kind of demonstrating that there is a good faith engagement in this kind of debates.

[00:38:19] Yes, why do all of these grandmas in Pensacola care about data centers in Texas? It's a great question, sometimes it's confusing. Okay, so I've got a big question for you. I think it has multiple levels. Susie from DC is asking, what should the average person be on the lookout for when they're on social media this election year? So that's a great question. Let's do a stack and please talk about what the average person can do and also what can governments and other institutions do? What can we, now that we know these influence operations are being run on us, people are confused, people are fighting more, what should people do?

[00:38:58] So I've got a good analogy for kind of how I think about the information environment and maybe the way you should think about your information diet is the same way you consume kind of like your own nutritional diet, right? In the sense that if I walked into the street and I saw a piece of steak on the road, like I shouldn't just eat that, right? I should probably ask the questions around where did this steak come from? How long has this steak been here for? Who's paying for the steak to be here in front of me, right? I feel, I think, it's the same kind of analogy should apply to kind of what you see on social media or what you see in public debates, right? To really understand kind of like, okay, why am I getting a certain piece of information? Who's kind of like spreading it? What are their intentions? I'm particularly motivated by this because I'm sort of moving to Japan and in Japan, all the supermarkets, the steak tells you which prefecture the steak's come from. So being able to trace back.

[00:39:44] Speaker A: from. So being able to trace back where the source of information, I think is really important as a way to sort of understand a broader context around that particular debate and who's maybe contributing to it as well. I mean, what is social media but a bunch of steaks on the road? I don't know. It sounds like a lot of work to figure out where they came from.

[00:40:04] Speaker B: Yeah. But I mean, people spend a lot of time, for example, I've curated a list of my favorite restaurants in DC and in San Francisco. I think people put the same amount of effort curating their own personal diets and same way they curated maybe their favorite journalists or their favorite media outlets and make sure they had a diverse diet of information as well. I think that hopefully should produce stronger people in terms of their ability to consume information.

[00:40:24] Speaker A: So don't just be going out, reading a bunch of strangers, like kind of figure, get some long term relationships, even if they're just online, figure out who's saying things that you trust. Is that what you're saying?

[00:40:35] Speaker B: Exactly right. And find people that you have similar tastes to and are credible, right, who have demonstrated, kind of, in the same way, you trust certain brands or certain food groups because they have health standards that means the food you're getting from them is kind of on high standards and safe. Same thing applies to the information space too, right? You want to trust people who are giving you kind of quality information too.

[00:40:56] Speaker A: Yeah. Yeah. Ben, what do you think in the stack of individuals up through institutions, what do you think people can do?

[00:41:08] Speaker C: I mean, so the thing that I've been studying in influence operations for over 10 years now and the thing I've seen from operations time and again all around the world in many different contexts, they have the commonality that the kind of content they're posting looks like it's designed to make people angry or afraid. It's targeting emotion far more often than getting a straight up sort of false claim you'll get a post which looks like it's meant to trigger people, make them really scared, make them really angry. Because when people are in that heightened emotional state they're much easier to manipulate. You know what it's like when you're really mad about something, it's much easier to make a bad decision.

[00:41:48] Speaker C: And something that we've consistently see from influence operations is that they'll hit on the emotion, they will try and trigger rage or fear. And so for anybody out there, if you're reading social media, if you're scrolling and you see a post which makes that little center in your brain explode with tension or fear or alarm or anger, take a step back and think, why is somebody trying to manipulate me here? What are they trying to manipulate me into? I mean, it could be an influence operation, it could be a scam, we don't know without looking more into it. You won't know without looking more into it.

[00:42:25] Speaker C: But it's the emotion that gets targeted. And so if you see an emotional post, try to just step back on it and think through, why is somebody trying to influence me here into doing something and what are they trying to do? Maybe they just want to get clicks. But if what they're doing is targeting your emotions, then they're putting you in danger of them making a wrong decision. So just take that step back and think through, why is somebody trying to mess with me here?

[00:42:44] Speaker A: So just to connect the dots, we've got high level strategy in Beijing, five-year plan, got to get ahead on AI. We've got government officials, paid money to influence American opinion and they're saying to Chatsypt and other tools. They're saying, help me pretend to be somebody in Pensacola and also make it scary. And then they're posting that on a social media that's coming to folks in America who actually really do care where this country goes. And then they have legitimate opinions, but those people are getting influenced.

[00:43:28] Speaker C: They're being emotionally manipulated by these operations. They could be emotionally manipulated by these operations if they see the operations at all, if the operations break through and if the operations don't get caught. Now, luckily in this case, the operations got caught and they didn't have any impact before we took them down. But what the operations and what the operators are trying to do is get to that space when they can rile people up or scare them because then you can steer them in the direction you want. And our job is to catch them doing it early, take them down, expose them, do everything we can to make sure they can't do that.

[00:44:00] Speaker A: Yeah, well, I'm glad we have you guys and your team. Thank you for your work. This has been a great talk. I've learned a lot. I feel like there are a lot more questions out there and a lot more that I could ask. Spamoflosh, who knew? So this was a great event. Thank you everybody in the community for joining us. I hope you learned as much as I did. I thought it was fascinating. We're going to have a lot more events coming over the summer. You're going to get emails soon telling you what they are, and I hope to see you there. So thank you for joining us today.

[00:44:42] Thank you very much.