Artificial intelligence in the current threat environment
Artificial intelligence has shifted from an emerging technology discussed in strategy papers into operational infrastructure that both attackers and defenders use every day. The change is not theoretical. Australian organisations are encountering AI-enabled threats in real incidents, and the security industry is racing to match adversarial capability with defensive tools that can operate at the same speed. Understanding what AI actually does to the threat landscape, on both sides of that line, is now a practical requirement for any organisation that takes security seriously.
How AI is changing the threat landscape
The Australian Signals Directorate’s Annual Cyber Threat Report 2024-25 states that AI is “almost certainly” enabling malicious actors to launch attacks at a greater scale and speed than was previously possible (https://www.cyber.gov.au/about-us/view-all-content/reports-and-statistics/annual-cyber-threat-report-2024-2025). Phishing was recorded in 60 per cent of incidents reported to ASD’s Australian Cyber Security Centre in the 2024-25 financial year, a figure that reflects how productive automated and AI-assisted social engineering has become as an entry vector. The same report notes that generative AI is enabling cybercriminals to scale deception through convincing deepfakes, fabricated identity verification records, and hyper-personalised phishing content that would previously have required significant manual effort to produce.
What has changed is not the existence of these techniques but their accessibility and throughput. Crafting a convincing spear-phishing email once required an attacker to research a target, write in that target’s language, and replicate plausible organisational context. Generative AI handles that work at scale. A campaign that previously targeted dozens of individuals can now target thousands with the same level of apparent personalisation. The quality threshold has also risen: AI-generated content no longer carries the grammatical errors and cultural mismatches that trained users learned to spot.
How adversaries are using AI
The clearest documented example of AI-enabled fraud in recent memory is the February 2024 attack on engineering firm Arup, in which an employee transferred approximately 25 million US dollars after attending what appeared to be a legitimate video conference with colleagues and senior executives. Every participant in that call except the victim was an AI-generated deepfake (https://www.weforum.org/stories/2025/02/deepfake-ai-cybercrime-arup/). The attack required no exploitation of technical vulnerabilities. It exploited trust, authority, and the assumption that a video call with recognisable faces is a reliable form of identity verification. That assumption is no longer safe.
Voice cloning has become similarly accessible. In a widely reported case involving a UK energy company, attackers used synthesised audio to impersonate the CEO’s voice convincingly enough that an employee transferred approximately 243,000 US dollars to a fraudulent account. The voice model required only a small sample of real audio to produce a replica that passed scrutiny in a live telephone call. Tools capable of this are now commercially available and trivially cheap to operate.
Beyond social engineering, AI is being applied to technical attack stages. AI-assisted vulnerability discovery allows attackers to process large codebases and configuration files far faster than human researchers, identifying candidate vulnerabilities that can then be investigated and exploited. Automated reconnaissance tools use AI to profile targets by aggregating publicly available data across domains, constructing detailed pictures of an organisation’s personnel, technology stack, external attack surface, and supplier relationships. This reconnaissance is continuous, passive, and difficult to detect.
Malware development has also been affected. Large language models assist with code obfuscation, helping to rewrite malicious code in ways that defeat signature-based detection. Polymorphic malware, which mutates its own code to avoid recognition, has existed for years but was difficult to produce at scale. AI-assisted code generation lowers that barrier substantially. Security researchers have demonstrated that publicly available LLMs, with modest prompt engineering, will assist in writing functional malicious code, even when the models include nominal safety filters.
How defenders are using AI
Security operations centres receive volumes of telemetry that human analysts cannot practically review in full. AI-driven anomaly detection systems work across that data continuously, identifying statistical deviations from baseline behaviour that would otherwise be invisible until an incident was already underway. Behaviour-based endpoint detection products use machine learning models trained on normal activity patterns to flag processes, network connections, and file operations that fall outside expected parameters, without relying on known-bad signatures that attackers can avoid.
Threat intelligence synthesis is another area where AI has materially improved defensive capacity. Analysts previously spent significant time manually correlating indicators across reports, feeds, and internal telemetry. LLM-assisted platforms can now process and summarise large volumes of threat intelligence in structured formats, accelerating the time from raw indicator to actionable context. This is not a replacement for human analysis but it removes a substantial amount of low-value processing work and allows analysts to focus on judgement-intensive tasks.
AI-assisted incident response is also maturing. During an active incident, AI tools can help responders rapidly map lateral movement across a network, correlate authentication events, identify data staged for exfiltration, and prioritise containment actions based on asset criticality. The same capability that allows attackers to move faster through a network can, when deployed defensively, help responders understand scope and act faster than a purely manual investigation would allow.
SOC automation platforms now use AI to triage alerts, suppress known-benign events, and route genuine detections to the appropriate response workflow without requiring human review of every individual alert. This addresses one of the most persistent operational problems in security: alert fatigue, where the volume of false positives erodes analyst attention and increases the probability that real events are missed.
The Australian government’s position on responsible AI
The Australian Government has approached AI governance through a suite of interconnected policies. The Policy for the Responsible Use of AI in Government, version 2.0, came into effect on 15 December 2025 and applies to all non-corporate Commonwealth entities. It requires agencies to develop a strategic approach to adopting AI, establish processes for operationalising responsible use, designate clear accountability for AI use cases, and undertake risk-based assessments at the use case level (https://www.digital.gov.au/ai/ai-in-government-policy). The policy emerged from work by the AI in Government Taskforce, co-led by the Digital Transformation Agency and the Department of Industry, Science and Resources.
The National Framework for the Assurance of Artificial Intelligence in Government, released in June 2024 by the Data and Digital Ministers Meeting, establishes practices of AI assurance applicable across Commonwealth and state agencies (https://www.finance.gov.au/government/public-data/data-and-digital-ministers-meeting/national-framework-assurance-artificial-intelligence-government). Separately, Australia’s AI Ethics Framework from the Department of Industry, Science and Resources articulates eight principles for responsible AI development and deployment, covering fairness, privacy protection, transparency, accountability, and contestability (https://www.industry.gov.au/publications/australias-artificial-intelligence-ethics-framework). These frameworks are not purely aspirational. They carry procurement and operational implications for government agencies and, increasingly, for vendors supplying AI-enabled products and services to government.
The ASD has also published guidance specifically addressing AI in security contexts, including joint guidance co-authored with the United States Cybersecurity and Infrastructure Security Agency, the National Security Agency, the FBI, and partner agencies across the Five Eyes. That guidance covers secure integration of AI in operational technology environments and addresses the risks of AI systems being manipulated, poisoned, or exploited by adversaries (https://www.cisa.gov/news-events/alerts/2024/01/23/cisa-joins-acsc-led-guidance-how-use-ai-systems-securely).
The dual-use problem
The same AI capabilities that make defensive tools more effective also make offensive tools more capable. This is not a future concern. Penetration testers and red teams are already using AI assistants to accelerate their work, and the same tools are available to malicious actors without restriction. An AI system trained to identify anomalies in network traffic can, with different framing, be used to identify which anomalies are most likely to go undetected. A model that summarises threat intelligence for defenders can summarise an organisation’s public attack surface for an attacker.
This dual-use reality has practical implications for how organisations think about AI adoption in security contexts. AI tools introduced into a security environment expand the attack surface if they are not themselves properly secured. Adversarial inputs, model poisoning, and prompt injection attacks against AI security tools are documented threat vectors. An AI-driven detection system that can be manipulated into ignoring specific activity patterns provides a false sense of security that is, in some respects, worse than no detection at all.
Organisations adopting AI-driven security capabilities need to treat the AI systems themselves as assets requiring protection, not just as protective infrastructure. That means understanding where models are hosted, what data they are trained on, who can query them and with what inputs, and how the organisation would detect manipulation of model behaviour. The ASD’s guidance on securing AI systems provides a starting framework for these questions (https://www.cyber.gov.au/business-government/secure-design/artificial-intelligence).
Where this leaves Australian organisations
AI has made some categories of attack significantly cheaper and more scalable. It has also made some categories of detection significantly faster and more consistent. The net effect is not a simple shift in either direction but an acceleration of the overall tempo of the contest between attackers and defenders. Organisations that adopt defensive AI capabilities thoughtfully, secure those systems properly, and maintain skilled human oversight of automated processes will be better positioned than those that either ignore AI entirely or deploy it without critical assessment of its limitations and risks.
The technical controls, governance frameworks, and staff awareness programmes that Australian organisations already maintain remain relevant. AI does not make them obsolete. It raises the baseline capability of adversaries, which means the quality and currency of defensive practice needs to rise to match. The ASD’s ACSC publishes guidance and alerts addressing AI-related threats as they evolve, and monitoring that guidance should be part of any organisation’s standard security practice (https://www.cyber.gov.au/about-us/view-all-content/reports-and-statistics/annual-cyber-threat-report-2024-2025).