A picture is not necessarily the truth! Uncovering the truth about rumors created using AI technology

Time:Oct 22,2024

In recent years, with the increasing popularity of artificial intelligence technology, the cost of fabricating and spreading rumors online has also become lower. Using AI synthesis technology to fabricate rumors, and even generate images and videos that are close to reality, has subverted the conventional understanding of "a picture is worth a thousand words," making it difficult to distinguish between truth and falsehood.

In recent years, with the increasing popularity of artificial intelligence technology, the cost of fabricating and spreading rumors online has become lower. Using AI synthesis technology to fabricate rumors, or even generate images and videos that are close to reality, has subverted the conventional understanding of "a picture is worth a thousand words," making it difficult to distinguish between truth and falsehood.

In January of this year, a message about an "explosion in Xi'an" appeared on a certain online platform. The message claimed that at 11:48 pm on January 10, a huge explosion suddenly rang out in the city. The text was also accompanied by a picture of the alleged explosion. This sensational news quickly spread online, even including key information such as time and location, making it extremely deceptive. So what is the truth?

The police investigated and found that the account that posted the message belonged to an MCN agency in Nanchang, Jiangxi Province. The actual controller of this company is Wang Moumou.

Li Yi, Captain of the Network Security and Protection Team of the Gaoxin Branch of the Nanchang City Public Security Bureau in Jiangxi Province: We received a notification from the Xi'an police that an MCN agency in our jurisdiction had posted information about an explosion in Xi'an. After local verification, no such incident occurred. Under these circumstances, we initially determined that the information released by the MCN agency involved online rumor-mongering and was false information. We immediately summoned the company's responsible person for questioning, inspected the company's office, and collected evidence from the computers and software tools used on-site.

During the questioning, the police learned from Wang Moumou that the rumor about the explosion in Xi'an, from text to images, was entirely AI-generated.

Li Yi, Captain of the Network Security and Protection Team of the Gaoxin Branch of the Nanchang City Public Security Bureau in Jiangxi Province: The offender Wang Moumou used a domestic website that provides recently generated hot news reports, or things that are happening or about to happen, providing writers with key keywords in a graded and classified manner. Using this website to generate keywords formed the article's title.

Wang Moumou, using AI software with voice interaction capabilities, only needed to speak keywords, and the software would automatically collect relevant information from the internet, generating hundreds to thousands of words of text and accompanying it with fake images that seemed relevant to the event, thus forming content that seemed information-rich but was actually fabricated. The offender Wang Moumou used another software to automatically upload this AI-generated content to multiple accounts on different online platforms controlled by the MCN agency and released it to the public.

Li Yi, Captain of the Network Security and Protection Team of the Gaoxin Branch of the Nanchang City Public Security Bureau in Jiangxi Province: From the perspective, direction, and tone of a news writer, the AI software was instructed to generate articles and news-related images using keywords such as "explosion" and "livelihood." The entire process required virtually no human intervention, resulting in the creation of this false information.

Using AI software to generate content, this MCN agency generated a large number of articles in a short period, with a peak of 4,000 to 7,000 articles per day. On March 28 this year, the police summoned Wang Moumou. Wang Moumou confessed to fabricating a large number of rumors using AI software. According to Wang Moumou's confession, the main purpose of generating these articles was to increase the likes and attention on the accounts they operated, thereby earning traffic.

During the investigation, the police discovered that the content released by the MCN agency online mainly focused on livelihood information. Most of it seemed closely related to people's lives, but it would add exaggerated plots, sensational elements, or fabricate false information to gain the attention and likes of netizens, thereby achieving traffic monetization.

Wang Moumou posted the AI-automatically generated articles on the platform, and the platform paid them accordingly based on the number of likes and views of the articles. Through this method, Wang Moumou received high remuneration in a short period.

Li Yi, Captain of the Network Security and Protection Team of the Gaoxin Branch of the Nanchang City Public Security Bureau in Jiangxi Province: After Wang Moumou generated the articles and pushed them to various platforms, based on the attention and views, for example, if the views reached 10,000, the coefficient is 1, then multiplied by attention, forwarding, and comments, and then multiplied by a coefficient of 1.5, the final remuneration paid by the platform is generated. If there is advertising revenue, it will be paid separately. They published 4,000 to 7,000 pieces of information per day using AI tools on average, with the highest single income being 700 yuan. Based on preliminary estimates, the daily income was over 10,000 yuan.

The police found that sensational content in the online rumors released by the MCN agency was often more eye-catching. For example, an article generated by Wang Moumou using AI about "the founder of a real estate company being taken into custody" received high online attention. According to the police, this piece of content also had higher income.

Li Yi, Captain of the Network Security and Protection Team of the Gaoxin Branch of the Nanchang City Public Security Bureau in Jiangxi Province: Because this information is considered "hot news," and the article's story and sensationalism are also relatively strong, it attracted high social attention.

According to the police investigation, Wang Moumou operated 5 MCN agencies and 842 accounts. Since January of this year, one of the MCN agencies has used its accounts to continuously release multiple false information pieces on a certain platform, such as "an explosion occurred in Xi'an on January 10, 2024, resulting in the destruction of buildings and vehicles, and casualties."

Li Yi, Captain of the Network Security and Protection Team of the Gaoxin Branch of the Nanchang City Public Security Bureau in Jiangxi Province: They were all on the same platform, using different names or types to publish information of different categories that people are interested in, such as livelihood and social hot topics.

It was found that Wang Moumou's actions constituted the spread of rumors and the fabrication of facts to disrupt public order. Currently, the Nanchang police have, in accordance with the relevant provisions of the People's Republic of China's Public Security Administration Punishment Law, legally imposed administrative detention for 5 days on Wang Moumou and ordered the relevant MCN agency to suspend operations for rectification.

AI-generated rumors that mimic news style are more deceptive.

Using AI technology to mimic the style of news reports, one-click generated rumor information is more deceptive. False news involving danger or disasters can obtain higher traffic. In Chongqing, public security organs investigated a case of an individual using AI software to spread rumors. The fabricated rumors of a residential building explosion led him to believe that he could achieve traffic monetization. This case was listed by the Ministry of Public Security as a typical case of combating and rectifying illegal and criminal activities related to online rumors.

04:10

On December 25, 2023, news of a "residential building explosion accident in Wuxi, Chongqing, resulting in 4 unfortunate deaths, and the authorities urgently intervening in the investigation" was posted on a certain online platform. The full text, over 500 words, used the tone of a news report to provide details such as "a sudden explosion accident in Wuxi County, Chongqing, with a very tragic scene," "4 victims were found, including a child," and "suspected to be related to residents' illegal wiring; the investigation team will further collect evidence." In a short time, this so-called "breaking news" received tens of thousands of clicks. The soaring number of views made Kang Moumou, who previously had no traffic on his online posts, somewhat excited at first.

Kang Moumou, the offender: At that time, I posted more than 20 posts without any income. Later, I saw a news report about an explosion with tens of thousands of likes, so I inputted "Chongqing explosion," and it automatically generated the content. I then copied its title and article and published it.

In this case, an individual surnamed Kang, noticing the high attention given to a news report about an explosion, used an AI text generation software to input the keywords "Chongqing" and "explosion," generating the aforementioned unfounded and fabricated false "news" and posting it on his online account.

The reason Kang fabricated the rumor was because he learned that creating and publishing articles on a well-known online platform could generate revenue based on the number of reads, comments, and shares, with a reward of 1 to 2 yuan per thousand effective reads. To earn money, after seeing that an old news report about an explosion had a huge number of clicks, he thought of fabricating false news to attract traffic. However, this action violated the law.

Kang: By 7 pm, the article had tens of thousands of views, and thousands of effective reads, then the police called me.

Chen Peng, Chongqing Municipal Public Security Bureau Cyber Security Team Police Officer: After our verification, this was a complete fabrication. Because the impact was quite severe, we reported the situation to the local police, Liangping District Public Security Bureau, to hold the rumor-monger accountable.

When the police found Kang, he was still immersed in the joy of finally creating viral content, having no idea of the legal consequences he might face.

Mao Yuncan, Chongqing Liangping District Public Security Bureau Longmen Police Station Police Officer: Kang admitted to using AI tools to generate a false article and posting it online. However, Kang clearly did not realize that his actions had violated relevant laws. He thought that as long as he deleted the article, nothing would happen, but in fact, his actions would have corresponding legal consequences.

To increase the exposure and views of his account, Kang unscrupulously exploited online hot topics, fabricating false news to obtain platform rewards and monetize traffic. According to police investigation, Kang confessed to his illegal act of fabricating facts to disrupt public order.

Mao Yuncan, Chongqing Liangping District Public Security Bureau Longmen Police Station Police Officer: Kang's actions violated the provisions of Article 25 of the Public Security Administration Punishment Law. The public security organs legally imposed a three-day administrative detention on him, confiscated his illegal gains, and banned his online account.

Chen Peng, Chongqing Municipal Public Security Bureau Cyber Security Team Police Officer: In the era of internet technology, current AI technology can not only generate text but also generate images and videos that are close to reality. This completely subverts our conventional understanding of "a picture is worth a thousand words," making it difficult to discern the authenticity of online rumors and increasing their social harm. The internet is not a lawless place. For those who publish false statements online and disrupt social order, the public security organs will resolutely and legally deal with them. For those who cause serious harm and have serious circumstances, the public security organs will pursue their legal responsibility according to the law.

Well-designed script, mutual verbal abuse for traffic generation, then live-streamed fight

The unscrupulous malicious hype for traffic and profit seriously disrupted social order. In Zhengzhou, Henan, two online streamers pre-designed a script, organized people to verbally abuse each other to attract traffic, and then live-streamed a fight. Just as the two groups were about to meet, they were stopped and controlled by the police.

01:50

On the evening of June 8, at around 10 pm, the Zhengzhou Municipal Public Security Bureau Zhengdong New District Branch received a report that a blogger nicknamed "Lao Mou" was "arranging a fight" online with a blogger nicknamed "Jiang Mou" on an online live-streaming platform.

Xu Hang, instructor of the Zhengzhou Municipal Public Security Bureau Zhengdong New District Branch Business District Police Station: After receiving the report, we immediately went to the live broadcast room, searched for the location of these people, and dispatched police officers to control them on the spot.

The police learned that the two "fighting" streamers had hundreds of thousands of fans on the online platform, and their usual source of income was through online live streaming. The so-called "fight" was actually a script they had planned in advance. That day, they organized people to verbally abuse each other, inciting antagonistic emotions among fans, and released videos to preheat and attract traffic. Then, both sides started an offline "fight" through online live streaming, attracting 78,000 viewers in an hour and a half, causing a negative social impact.

Xu Hang, instructor of the Zhengzhou Municipal Public Security Bureau Zhengdong New District Branch Business District Police Station: At that time, the total number of clicks was over 100,000. Because their broadcast time was not long, just over an hour, the gifts received were about 2,000 yuan. When we entered the live broadcast room, we saw that there were more than 8,000 people online at the same time, which had a very bad impact online.

In this case, in addition to the two online streamers, eight other men were involved, the oldest being 29 years old and the youngest 19. Currently, the ten suspects have been criminally detained by the police for suspected of provoking trouble.

Zhang Xiaofu, Zhengzhou Municipal Public Security Bureau Internet Supervision Detachment Police Officer: Their unscrupulous malicious hype for traffic and profit seriously disrupted social order, damaged the online environment, and affected the values of young people, and must be resolutely and legally cracked down on. Here, we would like to remind online streamers that cyberspace is not a lawless place. As online streamers, they should set a good example, actively spread positive energy, and use their positive influence to guide fans to form correct values. Unscrupulous malicious hype for traffic will ultimately backfire.

Legal experts: Platforms should fulfill their corresponding supervisory responsibilities

Whether it is using AI to fabricate shocking false news or live-streaming verbal abuse, the purpose is to earn platform traffic subsidies and monetize traffic. Legal experts said that it is sometimes difficult for ordinary netizens to directly judge whether a large amount of false information is a rumor, and platforms should fulfill their supervisory responsibilities for relevant information.

01:50

Chen Wei, professor at the School of Law, Southwest University of Political Science and Law: If the platform cannot fulfill this responsibility, it will be particularly difficult for the general public to filter information. Relevant regulations have also established that platform operators should fulfill their corresponding supervisory responsibilities and prevent the subsequent impact of such bad or false information being uploaded to the Internet.

Some online platforms pay rewards to high-traffic content and users, which to some extent stimulates some online users who want to make money to publish false information. Legal experts said that while stimulating creation, platforms should also take more responsibility to eliminate the spread of false information.

Chen Wei, professor at the School of Law, Southwest University of Political Science and Law: The payment of rewards should also play a positive guiding role and should not increase traffic by publishing false information. Once such disseminators appear, the relevant punishment mechanisms, or the subsequent warnings, account bans, and other methods should be considered.

It is easy to spread rumors, but difficult to debunk them. Is the cost paid by rumor-mongers too low, leading to the continuous proliferation of rumors? Legal experts suggest that the law should make rumor-mongers pay a higher cost.

Professor Chen Wei of the Law School of Southwest University of Political Science and Law: The cost of illegality must be reflected. In administrative penalties, not only administrative detention is involved, but also administrative fines, and measures such as account bans. We must keep up with these measures in a timely manner to match the proportion between the cost of illegality and the ultimate responsibility, to prevent others from seeing such a low cost and thus limiting the deterrent effect on the perpetrator.

 

Key word: A picture is not necessarily the truth! Uncovering the truth about rumors created using AI technology