The Dark Side of ChatGPT: How Criminals are Using Large Language Models for Cyber-Crime

For many of us, ChatGPT has quickly become our go-to tool for writing that tricky email or planning a dream vacation. Unfortunately, the dark side of this new technology is that criminals are also enthusiastically embracing AI bots – but for illicit purposes.   

One indication of criminals’ interest can be seen in a report by NordVPN, which found that the volume of posts related to ChatGPT in dark web forums increased by 145% from January to February 2023, including posts discussing how the bot’s sophisticated NLP abilities could be used for spear-phishing campaigns and social engineering.

Not only are criminals using AI bots developed by legitimate tech companies such as Google, Microsoft, and OpenAI, but there is also a worrying trend of new ‘dark’ AI bots emerging, such as WormGPT and FraudGPT, which have been developed expressly for malicious uses. What these AI bots, both legitimate and malicious, have in common is that they are all powered by large language models (LLMs). Read on to learn what law enforcement authorities must know to stay one step ahead. 

What are large language models? 

LLMs are AI models that utilize deep learning techniques to understand and generate human language. These models are typically trained on vast amounts of text to learn statistical patterns, relationships, and language structure within the data. LLMs excel at a variety of natural language processing (NLP) tasks, including text completion, sentiment analysis, language translation, and question answering. They leverage generative AI techniques to generate coherent and contextually relevant text, mimicking human language patterns. 

ChatGPT is currently the most well-known and widely used AI bot and is powered by the GPT-3 and GPT-4 models, which were developed by OpenAI. Other popular AI bots include Microsoft Bing and Google Bard, as well as a wide range of LLMs like BERT, PaLM 2, LaMDA, and others. 

The dark side of ChatGPT: leveraging large LLMs for cyber crime

ChatGPT and other AI bots developed by legitimate companies include safety features designed to prevent malicious usage. Many of these safeguards, however, can be easily circumvented through prompt engineering.

Bad actors can leverage LLMs to accelerate the speed and scale at which they commit illicit activities, including:

• Cyber-crime – LLMs can enable criminals to conduct fraud and create malware faster and more effectively.
• Financial crime – LLMs could be leveraged to effortlessly create false documentation on a mass scale to facilitate financial crimes. This could include generating fake supporting documents, customer profiles, transaction records, financial statements, invoices and contracts.
• Human trafficking – Traffickers often use social media platforms and messaging apps to recruit, groom, and exploit their victims. LLMs could be exploited to generate more effective text and content, and to create it faster, which would be leveraged to reach more victims.
• Creating and spreading disinformation and fake news – LLMs can enable faster and more widespread generation of fabricated news articles, false or misleading information, and inflammatory content.

We can be sure that as LLM technology evolves, criminals will find a way to leverage it to accelerate other types of criminal activities as well.

Accelerating cyber fraud and scams with large language models

Both the FBI and Europol have recently sounded the alarm that criminals can utilize generative AI and LLMs in numerous ways to accelerate cybercrime activities: 

LLMs can enable more effective fraud, social engineering and impersonation to scam victims into providing personal information, credentials or payments. In the past, basic phishing scams were relatively easy to identify due to grammatical and linguistic errors.

However, it is now possible to create more realistic impersonations of organizations or individuals, even with only a basic understanding of the English language. Additionally, LLMs can be exploited for various forms of online fraud, such as creating fictitious engagement on social media platforms, including promoting fraudulent investment proposals.  

LLMs provide the ability to create more believable and seemingly legitimate phishing emails, and to do so at scale. By automating the creation of highly convincing fake emails, personalized to the recipient, criminals can increase the chances of success for their attacks. In addition to quickly generating text based on simple prompts, LLMs can utilize original documentation, such as emails from a legitimate payment provider, to compose content that mimics the genuine author’s writing style, tone, and commonly used words.

Criminals can also leverage LLMs to run automated bots, which can convincingly mimic the communication style of real individuals or customer service representatives, to aid scams relating to phishing emails or malicious websites.  Bad actors have begun using LLMs to scam individuals out of money or data in “love scams.” LLMs are especially useful for scammers who are not fluent in the language in which they are trying to conduct the scam. In a recent example published by Sophos News, a scammer neglected to delete text indicating that the content was generated by a bot from a message sent to an intended victim (see below).

Creating customized malware with large language models

Experts are already seeing the first instances of cybercriminals using ChatGPT to develop malware. ChatGPT lowers the bar for would-be hackers to try their hand at building effective malware and makes it easier for those without extensive development expertise or coding skills to develop much more advanced programs.

ChatGPT makes writing code in less common languages simpler for novices and experts alike. Anti-malware software tools have a harder time detecting malware in these languages, making it more likely to slip through the cracks of that security layer. The cybersecurity vendor Checkpoint recently demonstrated that ChatGPT can successfully conduct a full infection flow, from creating a convincing spear-phishing email to running a reverse shell-attack. According to a recent survey conducted by Blackberry, 51% of IT decision makers believe there will be a successful cyberattack credited to ChatGPT within the coming 12 months.  

The emergence of ‘dark’ AI bots

In recent weeks, two new AI bots powered by LLMs, WormGPT and FraudGPT, have been advertised on dark web forums.  

WormGPT, which was released in early July, was first spotted by cybersecurity researcher Daniel Kelley. WormGPT is reportedly based on GPT-J, an LLM developed by the research group EleutherAI in 2021. This version of the model lacks safeguards, meaning it won’t hesitate to answer questions that GPT-J might normally refuse – specifically those related to hacking. WormGPT licenses are being sold for prices ranging from €500 – €5,000 on HackForums, a popular marketplace for cybercrime tools and services. 

FraudGPT is exclusively targeted for offensive purposes, such as crafting spear phishing emails, creating cracking tools and carding, and was spotted by the Netenrich threat research team circulating on dark web marketplaces and Telegram channels in July 2023. The creator of FraudGPT claims the bot can create undetectable malware, find leaks and vulnerabilities, as well as craft text for use in scams. 

How can law enforcement stay ahead of criminals using large language models?

As LLM technology advances, it is crucial for law enforcement to remain at the cutting edge of these developments. Taking a proactive approach will allow them to anticipate and prevent misuse by criminals.  

In addition, LLM technology can be incorporated into software solutions that support law enforcement investigations and intelligence analysis, including network intelligence systems, decision intelligence platforms, cyber threat intelligence tools and more. An LLM powered ‘co-pilot’ can offer numerous benefits allowing analysts and investigators to conduct their work faster and more effectively, including:

• Enabling analysts and investigators to ask questions in natural language, rather than creating complex queries
• Providing contextual investigation-specific information
• Generating descriptive, diagnostic and predictive analytical insights
• Suggesting recommended actions and next steps

Read our in-depth report to learn more about how LLMs will impact law enforcement, as well as how this technology can help analysts and investigators conduct more effective investigations.