In 2020, when the South Korean government was pursuing a blackmail ring forcing young women to make pornographic videos for pay, they found something else floating around in the dark corners of social media: pornographic images with other people’s faces crudely tagged on.
They didn’t know what to do with these early attempts at deepfake porn. Eventually, Congress passed a vague law against those who make and distribute deepfake porn. But that didn’t stop the wave of AI-powered crime that has now taken over the country. online culture discriminates against women to new depths.
Over the past two weeks, South Koreans have been shocked to discover that a growing number of young men and teenagers have been taking hundreds of photos on social media of their classmates, teachers, and military colleagues — mostly women and girls, including minors — and using them to create sexually exploitative images and video clips using deepfake apps.
They spread the material through chat rooms on the encrypted messaging service Telegram, some with as many as 220,000 members. Deepfakes typically combine the victim’s face with a body in a sexually explicit pose taken from a pornographic film. Investigators say the technology is so sophisticated that it is often difficult for the average person to tell that they are fake. As the country grapples with the threat, experts have noted that in South Korea, enthusiasm for new technologies can sometimes outweigh concerns about their ethical implications.
But for many women, these deepfake videos are just the latest online manifestation of the deeply ingrained misogyny in their country—a culture that has now spawned young men who find sharing sexually explicit images of women online a pastime.
“Korean society does not treat women as human beings,” said Lee Yu-jin, a student whose university is among hundreds of middle schools, high schools and colleges where students have been victimized. She asked why the government did not do more “before it became a digital culture of stealing friends’ photos and using them for sexual humiliation.”
Online sexual violence is a growing problem globally, but South Korea is at the forefront. Whether and how it can successfully tackle the deepfake problem will be watched by policymakers, school officials and law enforcement elsewhere.
The country has a dark side of sex crimes that occasionally surface. One South Korean has convicted operates one of the world’s largest websites for child sexual abuse imagery. A K-pop artist was convicted of facilitating prostitution through a nightclub. For years, police have struggled spycam porn. And the mastermind of the extortion ring was investigated in 2020. was sentenced to 40 years jailed for luring young women, including teenagers, into making videos that he sold online through a Telegram chat room.
The rise of easy-to-use deepfake technology has added a dangerous dimension to such forms of sexual violence: Victims often don’t know they’ve been victims until they receive an anonymous text message or a call from the police.
‘Slave’, ‘Toilet’, ‘Rags’
For one 30-year-old deepfake victim, whose name is being withheld to protect privacy, the attack began in 2021 with an anonymous message on Telegram that read: “Hello!”
Over the next few hours, a series of pornographic and deepfake images and video clips followed, featuring her face, taken from family trip photos she had posted on social media. Words like “slave,” “toilet” and “rags” were written on her body.
In April, she learned from police that two of her former classmates at Seoul National University were among those arrested. The male graduates of the prestigious university, along with accomplices, targeted dozens of women, including dozens of former Seoul National University students, with deepfake pornography. One of the arrested men was sentenced to five years in prison last month.
“I can’t think of any reason why they would treat me like that, except that I’m a woman,” she said. “Having people like them around me makes me lose faith in my fellow human beings.”
She said she has struggled with trauma since the attack, with her heart racing every time she receives a text message notification on her smartphone or an anonymous call.
South Korea, whose pop culture is exported around the world, has become the country most vulnerable to deepfake pornography. More than half of the deepfakes globally target South Koreans, and the majority of these deepfakes target singers and actors from the country, according to “The State of Deepfake in 2023,” A study published by US-based cybersecurity firm Security Hero. Leading K-pop companies have declared war on deepfakes, saying they are gathering evidence and threatening to sue those who create and distribute them.
Still, the problem is growing. South Korean police reported 297 deepfake sex crimes between January and July, compared with 156 cases in all of 2021, when such data was first collected.
It wasn’t until last month, when local media exposed a large amount of deepfake content on Telegram, that President Yoon Suk-yeol ordered his government to “root it out.” Critics of Yoon note that during his 2022 presidential campaign, he denied that there was systemic gender discrimination in South Korea and promised to abolish the Ministry of Gender Equality.
News reports of the rise of deepfakes this year have sparked panic among young women, many of whom have deleted selfies and other personal images from their social media accounts, fearing they will be used for deepfakes. Chung Jin-kwon, who was a middle school principal before taking up a role at the Seoul Metropolitan Office of Education last month, said his alma mater had discussed whether to remove student photos from the yearbook.
“Some teachers refused to display their photos there, replacing them with caricatures,” said Mr. Chung.
Young people in South Korea, one of the world’s most connected countries, become tech-savvy from a young age. But critics say the country’s school system is too focused on preparing them for the most important things. college entrance exam that they are not taught how to handle new technology in an ethical way.
“We produce machines that solve problems in exams,” Mr. Chung said. “They don’t learn values.”
A push for tougher laws
Kim Ji-hyun, a Seoul city official whose group has counseled 200 teens involved in digital sexual exploitation since 2019, said some boys have used deepfakes to get revenge on ex-girlfriends — and in some cases, girls have used them to alienate classmates. But many young people are initially drawn to deepfakes out of curiosity, Kim said.
Chat room operators lure them with incentives, including Starbucks coupons, and ask them to provide photos and personal information of women they know. Some Telegram channels, known as “rape and humiliation rooms,” target individuals or women from certain schools, said Park Seong-hye, a team leader at the government-funded Korea Women’s Human Rights Institute who investigates digital sex crimes and provides help to victims.
Under a 2020 law, people convicted of creating pornographic or abusive deepfakes with the intent to distribute them could face up to five years in prison. Those seeking to profit from distributing such content could face up to seven years in prison. But there is no law against buying, storing, or viewing deepfakes.
Investigators need court approval to secretly access deepfake chat rooms, and they can only do so to investigate reports of sexual abuse of minors. The process can also be slow.
“You find a chat room on a holiday, but by the time you get court approval, the chat room is gone,” said Hahm Young-ok, a senior investigator on online crimes at the National Police Agency.
The government has promised to push for stricter laws against buying or viewing deepfake videos of sexual exploitation. This month, police investigating the latest spate of deepfake videos said they had Seven male suspects arrestedincluding six teenagers.
Pornography is censored on South Korea’s internet, but people can bypass controls using virtual private networks and hard-to-enforce bans on social media channels. Police have pointed out that They can investigate whether Telegram has has facilitated deepfake sex crimes. Last month, Telegram founder Pavel Durov was arrested in France and charged with multiple crimes, including enabling the distribution of child sexual abuse material.
Telegram said in a statement that it “has been actively removing reported content from South Korea that violates its terms of service and will continue to do so.”
Meanwhile, governments are under pressure to force online platforms to do more to filter content like deepfake pornography.
“It’s time to choose between protecting the platforms and protecting our children and young people,” said Lee Soo-jung, a forensic psychology professor at Kyonggi University. “What we see happening in 2024 was predicted in 2020, but we didn’t do anything.”
