Latest news

AI and Red Teaming

Red teaming is still fairly young as far as cybersecurity disciplines go – most of us in this part of the industry have come in from being penetration testing consultants, or have some sort of background in IT security with a good mix of coding and scripting skills to develop tools. Our work often requires us to not only simulate the threat actors as closely as we can, but also manage the risks of our operations to avoid impacting our client’s business. This dichotomy of outcomes (simulating a threat actor who’s objective is to disrupt, whilst simultaneously trying not to disrupt) may seem confusing, but we also need to remember what red team is for. Its to help our clients test their detection and response capabilities. The objective of the red team is almost incidental – it merely sets a direction for the consultants to work towards whilst we determine what our clients can and cannot detect and what they do about it, if a detection occurs. That latter part is where the disruption is more likely to occur but even there, we can manage the risks.

So where does AI come into it? Well, we have all seen the news about AIs going to take over jobs in a number of fields, and red teaming is no different from the fears of this. The problem is, most AI systems these days are just really good guessers – I prefer to think of these things as almost expert systems, instead of a true intelligence. By that, I mean you can train them to be exceptional at specific tasks but if you go too broad with them, they really struggle. They don’t explain themselves; they can’t repeat steps identically particularly well; and they often forget or hallucinate critical elements when faced with large and complex tasks. A red team is a very large and complex series of tasks where forgetting or imagining critical steps will often lead to a poor outcome. Add into that mix, live environments and risk management, and the dangers of impacting a client become uncomfortably high. As a result, I have not yet met a single professional in this industry who would be happy to take the risk of letting a red team run entirely with AI, and I don’t see that changing any time soon.

However, I do see a future in which AIs help co-pilot red teams. By this I mean, that if the privacy concerns can be addressed, I can foresee a point where a private specialist red team LLM AI would be permitted to ingest data the red team acquires during an engagement (such as network mapping information, directory lists, active directory data, file contents, source code, etc.), and having it perform analysis on it. It can then provide suggestions on how the engagement can proceed. This would also have the added benefit of it being able to answer questions rapidly for the red team to help them consider additional attack paths, identify additional issues in the environment, and suggest additional things they could try. It could also quickly confirm if the red team had any interactions with systems within the client environment to deconflict if issues occur. In time I could even see this being an added real-time benefit for client control groups who would be able to interrogate the LLM for quicker results as to what the red team are doing and what has been identified to date.

AI is here now, and its evolving. We can’t really ignore it as it becomes a tool more and more used in everyday lives, and that means we need to find ways to make it work with the concerns we have. I personally feel that pushing them into smaller expert system roles is the right way forward, as this then allows them to fulfil the role of an assistant more fully. We also need to acknowledge that the public models have been trained unethically on source data taken without consent from authors and copyright holders. As their use grows, not only is there a considerable environmental impact, but I believe they will start to show strain in the near future. This is because, as the public further embraces these tools and uses them to generate new content, that AI generated content will also be absorbed by LLMs. This risks us entering a situation where the snake will eat its own tail and turn into the LLMs into an echo chamber, and we will see the quality of their output drop considerably. This will also likely be compounded by people losing critical thinking skills, which ultimately will harm us more than the AIs can help us.