A few months ago, former Inter Milan football club owner Massimo Moratti received a call from “Italian Defense Minister Guido Crosetto.” The caller asked for financial assistance to help rescue an Italian journalist. Because of the urgent situation, Moratti transferred nearly one million euros to the caller. A couple of days later, when Moratti contacted Guido Crosetto again to follow up, he was told Guido Crosetto had never called him for help and realised he had been scammed. The scammers used AI to generate a deepfake of the Italian Defense Minister’s voice and used his voice call to some well-known people, including fashion designer Giorgio Armani and Prada founder Patrizio Bertelli, to ask for financial assistance. Although the stolen money was eventually recovered, the incident was still shocking. (For more information, click here)

These scams not only target those high profile individuals, but also target ordinary people. In California, an elderly man, Anthony, was tricked into believing his son was in a serious accident and needed urgent bail money. The voice he heard on the phone was exactly like his son’s. Following instructions from scammers, Anthony transferred over twenty-five thousand dollars to scammers. When Anthony realised his son was safe, he had fallen victim to an AI-generated voice scam(For more information, click here). Those cases are not isolated incidents. In 2024, over 845,000 imposter scams were reported in America(For more information, click here).

These cases show us that scammers are increasingly using AI deepfake to imitate the voices and faces of people we trust and use them to carry out phone scams. How did they do that? And what does it mean for our security and society? In this post, we will discuss how AI generates deepfake voice and video, why it’s proliferating, and how unprepared our laws and platforms are. We will also discuss what can be done to address the threat without preventing innovation.

(Source: REUTERS)

How Do Deepfake Voice & Video Clones Work?

In the past, replicating someone’s voice or face from a short video or an audio clip was extremely difficult, mostly seen in movie special effects. However, the rapid advancements in AI make it easy. Everyone could use software like Reface, Voice.ai, and Murf.ai to replicate voices and faces. Those tools only need audio recordings of that person. Those tools use AI algorithms to analyse and model unique vocal characteristics, thereby generating new speech that sounds like the original speaker. This process is called Voice Conversion(Zhang et al., 2023). The more recordings it has, the more accurate the clone will be. The same principle applies to video deepfakes.

While most users use those tools just for entertainment, scammers find it brings a new opportunity. If there is a short clip of you talking online, scammers may use AI tools to copy your voice and face. All they need is a technician to optimise AI-generated content to make it more realistic. This significantly lowers the cost of committing fraud.

Importantly, these voices and faces generated by AI and after optimise are hard to differentiate from the real ones. There may be tiny flaws in some deepfakes, but if you are panicked, like when you hear your daughter or son crying for help in danger, you really can not catch tiny imperfections. Even video evidence can not be trusted because scammers could use an AI tool to generate a fake video of someone. In short, the traditional ways of verifying truth are unreliable because of this technology. We’re entering an era where seeing or hearing is no longer believing.

Gaps in Law and Governance

So, why don’t we simply ban AI-generated voices and videos? Because this technology is not harmful. For instance, in The Mandalorian, the producers use AI tools to successfully recreate the voices of young Luke Skywalker(For more information, click here). In other words, the problem is not the technology but its malicious use.

Thus, what we need to do is find a way to regulate and prevent misuse, but our legal and governance systems are lagging behind. In many places, there are no specific laws to regulate. Take the United States as an example. There is still no federal law regulating malicious deepfakes, only a few state laws have tried to cover this issue.  Tennessee passed the ELVIS Act, becoming the first state in the United States to regulate AI-generated voice and image clones(For more information, click here). As of October 2024, 20 states have enacted regulations targeting deepfakes in elections. However, the United States still lacks a federal law on deepfake technology (For more information, click here). Although two bills have been introduced in Congress, such as the DEEPFAKES Accountability Act and the NO FAKES Act (2024), they have not been passed into law so far. This means the policy has not caught up with the advancements of AI innovation.

It is evident that policymakers realise they need to build new rules for AI deepfakes, but they face a new problem: how to prevent malicious deepfakes while not obstructing their positive development. There is no perfect solution so far, but some attempts have been made, like requiring AI-generated content to carry a watermark. Another example is the European Union’s AI Act, which will ban AI that manipulates human behaviour to circumvent users’ free will, including deceiving someone with a fake call(For more information, click here). But we do not know how effective they will be.

In addition, technology companies and service providers also play a role in policy implementation. Social media platforms have different policies on deepfake content, mostly focused on election interference and non-consensual pornography. Those rules are not enough and are incomplete. Phone carriers and messaging applications have not implemented verification systems to distinguish AI-generated calls or messages.

Moreover, laws in just one country may not be enough because these scams may cross borders. A scammer could be in one country to target victims all over the world. Fake calls may route through many countries or the internet, which makes it hard for police to trace the scammer. Even if new laws get passed in one country, it is a significant challenge to enforce an anonymous scammer overseas.

All of this leads to one question: Who is responsible when scammers use AI deepfakes to commit fraud? Obviously, the scammers are responsible for their crimes. But what about the creators of the AI tools? What about the social platforms or websites where scammers catch the victims’ voices and faces? Responsibility is diffuse, and it is hard to assign. This makes it easy for every party to shirk responsibility.

Can AI Help Us Detect Deepfakes?

How about using AI to fight with AI? Can we use technology to identify those fakes? Researchers are now working on detection tools that can determine deepfakes(Zhang et al., 2023). However, AI keeps developing rapidly, which means researchers and their tools are always needed to catch up. Moreover, as mentioned earlier, those tools are not widely used where they are most needed. For example, phone carriers can not identify whether the calls are AI-generated deepfakes.

The problem is that even if those detection tools exist, they are not always practical for everyday use. For instance, using them in phone networks or social media platforms can be costly, and phone companies and social media platforms may refuse to use those tools.

Our Digital Defenses Are not Ready

Another reason those scams are so rampant is that our digital defences are not yet prepared. We are now in a world where personal data is everywhere, but there is a lack of protection for our data. Almost everyone, especially younger generations, has some recordings of their voice and images of their face online on social media platforms like Instagram, TikTok and YouTube. But we do not have any strong protections to protect our voices and faces. For example, we do not have default rights over how others can use our publicly shared voice or image. If we post our photos and videos online, our voices and faces seem to be essentially part of the public commons. Those data may become the training data of AI or stolen by scammers. This is a data governance problem: our infrastructure for sharing information, including the internet and social media, is not designed with deepfakes.

We could build a new system to protect users on the existing infrastructure and social media platforms. Just like when we fought back against phishing scams by developing spam filters and caller ID, we need similar measures to tackle deepfake scams. For example, phone companies could set up a special system to identify suspicious call patterns, such as a single number making multiple calls to different regions in a short time. Just like how banks can flag and block unusual credit card transactions. Or phone companies could use AI to analyse whether the voice is real during calls in real-time and alert the receiver if the voice seems AI-generated.

In conclusion, we need systemic solutions: a single rule or restriction for one specific area is not enough, and what we really need is a complete and organised approach. AI governance requires the involvement of multiple social forces rather than relying solely on the monopoly of a few companies(Crawford, 2021). This means it requires efforts from every part, including policy improvements, the development of detection tools, and involvement from phone companies.

Addressing the AI Scam Threat and Securing Our Digital Security

We need a comprehensive approach to effectively deal with the growing threat of AI deepfake scams and protect our digital safety.

First, governments must fill the gaps in policies and regulations on AI-generated deepfake audio and video. The international organisation should take the lead in establishing comprehensive rules. This is because transnational crime can only be avoided if the rules are broadly consistent around the world. Filling the gaps in only a few countries is ineffective because scammers may operate in one country and target victims in another country. In addition, it is crucial to enact new laws promptly to keep pace with the increasing number of scam cases reported.

In addition, governments should fund the protection of our digital security. The development of tools that could identify deepfake audio or video needs a lot of money, time and manpower. If the government can offer support for development, it may be fast. At the same time, governments also need to make sure those tools are widely accessible by phone carriers so that the deepfake audio or video can be easily identified.

The technology industry, including AI developers and social media platforms, must be part of the solution. Even though there may not be a perfect solution, those technology companies still have a duty to improve safety measures to protect their users. They must integrate ethical standards into their products, or they should be held accountable for the harm their products cause.

---

References

Crawford, K. (2021). Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence (pp. 1–21). Yale University Press.

Gibbs, A. (2024, March 21). TN Gov. Lee signs ELVIS Act into law in honky-tonk, protects musicians from AI abuses. Retrieved April 7, 2025, from The Tennessean website: https://www.tennessean.com/story/entertainment/music/2024/03/21/elvis-act-tennessee-gov-lee-signs-act-musicians-ai/73019388007/?utm_source=chatgpt.com

Lotz, A. (2025, March 15). AI voice-cloning scams: A persistent threat with limited guardrails. Retrieved from Axios website: https://www.axios.com/2025/03/15/ai-voice-cloning-consumer-scams?utm_source=chatgpt.com

News European Parliament. (2023, December 9). Artificial Intelligence Act: deal on comprehensive rules for trustworthy AI | News | European Parliament. Retrieved April 7, 2025, from Europa.eu website: https://www.europarl.europa.eu/news/en/press-room/20231206IPR15699/artificial-intelligence-act-deal-on-comprehensive-rules-for-trustworthy-ai?utm_source=chatgpt.com

Pollina, R. (2024, October 18). Scammers swindle elderly California man out of $25K by using AI voice technology to claim his son was in “horrible accident,” needed money for bail: “Absolutely his voice.” Retrieved from New York Post website: https://nypost.com/2024/10/18/us-news/scammers-swindle-elderly-california-man-out-of-25k-by-using-ai-voice-technology-to-claim-his-son-was-in-horrible-accident-needed-money-for-bail-absolutely-his-voice/

Public Citizen. (2024, October 16). Twenty States Enact Laws to Regulate Political Deepfakes  – Public Citizen. Retrieved April 7, 2025, from Public Citizen website: https://www.citizen.org/news/twenty-states-enact-laws-to-regulate-political-deepfakes/?utm_source=chatgpt.com

Respeecher. (2024, June 12). How Respeecher’s Voice Cloning Brought Young Luke Skywalker to Life in The Mandalorian. Retrieved April 7, 2025, from Respeecher.com website: https://www.respeecher.com/case-studies/respeecher-synthesized-younger-luke-skywalkers-voice-disneys-mandalorian?utm_source=chatgpt.com

Reuters. (2025, February 12). Italian police freeze cash from AI-voice scam that targeted business leaders. Retrieved from Reuters website: https://www.reuters.com/technology/artificial-intelligence/italian-police-freeze-cash-ai-voice-scam-that-targeted-business-leaders-2025-02-12/

Sumsub. (n.d.). Deepfake Cases Surge in Countries Holding 2024 Elections, Sumsub Research Shows. Retrieved from Sumsub website: https://sumsub.com/newsroom/deepfake-cases-surge-in-countries-holding-2024-elections-sumsub-research-shows/

Yi, J., Wang, C., Tao, J., Zhang, X., Zhang, C. Y., & Zhao, Y. (2023). Audio Deepfake Detection: A Survey. ArXiv (Cornell University), 14(8). https://doi.org/10.48550/arxiv.2308.14970