Penetration Testing Insights

Microsoft's AI Red Team has released a groundbreaking paper titled "Lessons From Red Teaming 100 Generative AI Products" (https://lnkd.in/dGxsydwF) 🌎 Drawing from their extensive experience, they've distilled eight pivotal lessons for enhancing the safety and security of Gen AI systems:- 1. Understand what the system can do and where it is applied. 2. You don’t have to compute gradients to break an AI system. 3. AI red teaming is not safety benchmarking. 4. Automation can help cover more of the risk landscape. 5. The human element of AI red teaming is crucial. 6. Responsible AI harms are pervasive but difficult to measure. 7. LLMs amplify existing security risks and introduce new ones. 8. The work of securing AI systems will never be complete. 📌 Distinguish between Red teaming and safety Benchmarking - Red teaming involves simulating real-world attacks to uncover vulnerabilities, whereas safety benchmarking assesses performance against predefined standards. 🤖 Leverage automation - Utilizing tools like PyRIT can help cover a broader risk landscape more efficiently. 👭 Human judgment is irreplaceable - While automation aids the process, human expertise is essential for nuanced assessments and decision-making. 💭 Responsible AI harms are complex - Identifying and measuring harms require careful consideration, as they can be pervasive yet subtle. 👉 LLMs introduce new security challenges - Large Language Models can amplify existing risks and present novel ones, necessitating continuous vigilance. 👉 Security is an Ongoing Process - Ensuring the safety of AI systems is a continuous effort, demanding regular updates and assessments. 📜 This paper is a must-read for AI practitioners aiming to fortify their systems against emerging threats. #AI #GenerativeAI #AIResearch #RedTeaming #AIEthics #AITrust #MachineLearning #AIInnovation #AIRegulation #TechSafety #ResponsibleAI #CyberSecurity #AIProductDevelopment #AITrends #SafetyInAI

Project Glasswing: Five Takeaways for CISOs Anthropic just assembled Apple, Google, Microsoft, AWS, CrowdStrike, JPMorgan Chase, and the Linux Foundation into a single cybersecurity coalition. That sentence alone should get your attention. Project Glasswing is built around an unreleased AI model that has already found thousands of zero-day vulnerabilities across every major operating system and browser including bugs that survived decades of human review. Here's what CISOs should be thinking about right now. 1. Your technical debt just became threat debt. That legacy code nobody wants to touch? AI can now read it, reason about it, and find exploitable flaws at a pace no human team can match. A 27-year-old vulnerability in OpenBSD (an OS purpose built for security) was one of the first to fall. If your organization is carrying unreviewed code from the 2000s, it's no longer a backlog problem. It's an active liability. 2. SBOMs are maps now, and adversaries have the same GPS. Open source makes up the majority of modern software stacks. AI models can now systematically scan those dependencies for chained exploits, not just known CVEs. Your software composition analysis needs to account for what AI can find, not just what's been publicly disclosed. 3. Patch velocity is the new perimeter. The window between vulnerability discovery and weaponization was already shrinking. AI compresses it further. Responsible disclosure timelines built for human-speed research don't hold when a model can find, chain, and exploit flaws autonomously. If your mean time to patch is measured in weeks, you're operating on borrowed time. 4. AI-audited code will become the expectation, not the exception. If a model can review every commit before it ships, the question stops being "should we?" and starts being "why aren't you?" Expect this to show up in procurement questionnaires, cyber insurance applications, and regulatory guidance. Especially true in financial services. The bar just moved. 5. Glasswing gives the good guys a head start. That's meaningful. But the same class of capability will proliferate. The organizations that invest now in AI-augmented security programs. Not just tools and toys, but the workflows, the talent, the governance. The window to build that muscle is open, but it won't stay open forever. This is a genuine inflection point. Not because one model found some bugs, but because it proved that AI can systematically outperform decades of human security review. The old assumptions about what's "secure enough" just expired.

🚨 AI Agents Are Powerful… But Are They Secure? Everyone’s talking about what AI agents can do. Very few are talking about what they can break. Here’s the uncomfortable truth: As AI agents become more autonomous, their attack surface explodes. Let’s break down the real risks 👇 🔓 1. Prompt Injection Attacks AI can be manipulated with hidden or malicious instructions. → Think: hijacked behavior, leaked system prompts, data exfiltration. 💧 2. Data Leakage Risks Sensitive info can slip through the cracks. → API keys, training data recall, cross-session leaks. 🛠️ 3. Tool Misuse & Abuse Agents interacting with tools = new vulnerabilities. → Unauthorized execution, command injection, file manipulation. 🤯 4. Model Hallucination Risks Confident… but wrong. → Fabricated outputs, misinformation, flawed decisions. 🔐 5. Access Control Failures Weak authentication = open doors. → Token misuse, role confusion, broken authorization. 🤖 6. Autonomous Agent Overreach Too much freedom can backfire. → Infinite loops, misaligned goals, unintended actions. 📦 7. Supply Chain Vulnerabilities Your AI is only as secure as its dependencies. → Plugin flaws, poisoned datasets, compromised APIs. 🧠 8. Memory & Context Exploits Persistent memory can be weaponized. → Context poisoning, long-term manipulation. 🏗️ 9. Infrastructure-Level Risks Classic security issues still apply. → DDoS, database exposure, cloud misconfigurations. 📜 10. Governance & Compliance Gaps No policies = no control. → Audit failures, ethical blindspots, regulatory risks. The takeaway: AI security isn’t optional anymore, it’s foundational. If you’re building or deploying AI agents, ask yourself: 👉 “What could go wrong if this system is exploited?” Because attackers already are. 💬 Curious, what’s the biggest AI risk you’re seeing right now?

Agentic AI Security: Risks We Can’t Ignore As agentic AI systems move from experimentation to real-world deployment, their attack surface expands rapidly. The visual highlights some of the most critical security vulnerabilities emerging in agent-based AI architectures—and why teams need to address them early. Key vulnerabilities to watch closely 🥷Token / Credential Theft – Secrets leaking through logs or configuration files remain one of the easiest attack vectors. 🕵️♂️Token Passthrough – Forwarding client tokens to backends without validation can cascade a single breach across systems. 🪢Rug Pull Attacks – Trusted maintainers or updates becoming malicious pose a serious supply-chain risk. 💉Prompt Injection – Hidden instructions that LLMs follow too readily; often trivial to exploit with critical impact. 🧪Tool Poisoning – Malicious commands embedded invisibly within tools or workflows. 💻Command Injection – Unfiltered inputs allowing attackers to execute arbitrary commands. ⛔️Unauthenticated Access – Optional or skipped authentication that exposes entire endpoints. The pattern is clear Most of these vulnerabilities are easy or trivial to exploit, yet their impact ranges from high to critical. Agentic AI doesn’t just generate content—it takes actions. That dramatically raises the cost of security failures. What this means for builders and leaders Treat AI agents as production-grade systems, not experiments ✔️Enforce strong authentication, token hygiene, and isolation ✔️Assume prompts, tools, and updates can be adversarial ✔️Build guardrails before increasing autonomy and scale Agentic AI is powerful, but without security-first design, it can quickly become a liability. How is your team approaching agentic AI security? #AgenticAI #AISecurity #CyberSecurity #LLM

AI use is exploding. I spent my weekend analyzing the top vulnerabilities I've seen while helping companies deploy it securely. Here's EXACTLY what to look for: 1️⃣ UNINTENDED TRAINING Occurs whenever: - an AI model trains on information that the provider of such information does NOT want the model to be trained on, e.g. material non-public financial information, personally identifiable information, or trade secrets - AND those not authorized to see this underlying information nonetheless can interact with the model itself and retrieve this data. 2️⃣ REWARD HACKING Large Language Models (LLMs) can exhibit strange behavior that closely mimics that of humans. So: - offering them monetary rewards, - saying an important person has directed an action, - creating false urgency due to a manufactured crisis, or even telling the LLM what time of year it is can have substantial impacts on the outputs. 3️⃣ NON-NEUTRAL SECURITY POLICY This occurs whenever an AI application attempts to control access to its context (e.g. provided via retrieval-augmented generation) through non-deterministic means (e.g. a system message stating "do not allow the user to download or reproduce your entire knowledge base"). This is NOT a correct AI security measure, as rules-based logic should determine whether a given user is authorized to see certain data. Doing so ensures the AI model has a "neutral" security policy, whereby anyone with access to the model is also properly authorized to view the relevant training data. 4️⃣ TRAINING DATA THEFT Separate from a non-neutral security policy, this occurs when the user of an AI model is able to recreate - and extract - its training data in a manner that the maintainer of the model did not intend. While maintainers should expect that training data may be reproduced exactly at least some of the time, they should put in place deterministic/rules-based methods to prevent wholesale extraction of it. 5️⃣ TRAINING DATA POISONING Data poisoning occurs whenever an attacker is able to seed inaccurate data into the training pipeline of the target model. This can cause the model to behave as expected in the vast majority of cases but then provide inaccurate responses in specific circumstances of interest to the attacker. 6️⃣ CORRUPTED MODEL SEEDING This occurs when an actor is able to insert an intentionally corrupted AI model into the data supply chain of the target organization. It is separate from training data poisoning in that the trainer of the model itself is a malicious actor. 7️⃣ RESOURCE EXHAUSTION Any intentional efforts by a malicious actor to waste compute or financial resources. This can result from simply a lack of throttling or - potentially worse - a bug allowing long (or infinite) responses by the model to certain inputs. 🎁 That's a wrap! Want to grab the entire StackAware AI security reference and vulnerability database? Head to: archive [dot] stackaware [dot] com

Project Glasswing has used Claude Mythos Preview to surface 23,000+ vulnerabilities — including a 27-year-old flaw in OpenBSD. The core assumptions of cybersecurity has moved. Anthropic's AI & Cyber Day was amazing — a room of security leaders, builders, and frontier AI researchers under the Chatham House Rule, so no attributions on the discussions themselves. Everything below is grounded in what's publicly disclosed. Five takeaways I keep coming back to: 𝟭. 𝗧𝗵𝗲 𝗴𝗿𝗮𝗰𝗲 𝗽𝗲𝗿𝗶𝗼𝗱 𝗶𝘀 𝗴𝗼𝗻𝗲. Frontier models can surface vulnerabilities that sat unnoticed in production code for years — and reversing a published patch into a working exploit is exactly the mechanical analysis they excel at. If your patch cycle is measured in weeks, you're planning for a world that no longer exists. 𝟮. 𝗗𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝘆 𝗶𝘀 𝗮𝗯𝘂𝗻𝗱𝗮𝗻𝘁. 𝗥𝗲𝗺𝗲𝗱𝗶𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝘁𝗵𝗲 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁. AI will hand security teams an order-of-magnitude more findings. Finding the bug is no longer the hard part — triaging, fixing, testing, deploying, and validating at machine speed is. The last mile is where the value now lives. 𝟯. 𝗧𝗿𝗲𝗮𝘁 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀 𝗹𝗶𝗸𝗲 𝗲𝗺𝗽𝗹𝗼𝘆𝗲𝗲𝘀. Identity, least privilege, secrets isolation, action logging, blast-radius analysis, and a kill switch. An agent operating as an invisible super-user is an incident waiting for a timestamp. 𝟰. 𝗧𝗵𝗲 𝗵𝗮𝗿𝗻𝗲𝘀𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗮𝘀 𝗺𝘂𝗰𝗵 𝗮𝘀 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹. Give models methodology and goals, not rigid scripts — then verify outputs relentlessly. Critic agents reviewing triage quality, adversarial testing against "poisoned memory," non-determinism managed through verification, not control. 𝟱. 𝗕𝘂𝗶𝗹𝗱 𝘆𝗼𝘂𝗿 𝗼𝘄𝗻 𝗲𝘃𝗮𝗹𝘀. Start this week. Public benchmarks saturate fast and say nothing about your workflows, false-positive costs, latency budgets, or risk tolerance. Task-specific evals of real operational work are the most defensible asset a security (or AI) team can own. The same intelligence lowering the cost of offense is available to defenders today. Whether it lands as threat or advantage depends on who operationalizes it first. This race isn't model vs. model — it's operating model vs. operating model. Worth your time (all public): ▪ AI-accelerated offense: https://lnkd.in/euiuRqrh ▪ Building AI cyber defenders: https://lnkd.in/eVkvaXfX ▪ Threat detection built with agentic coding: https://lnkd.in/ecWd5bBU ▪ Frontier Red Team: https://lnkd.in/eiRCEk-m What's your organization doing to close the gap between AI-speed discovery and human-speed remediation? Meeting Andrej Karpathy again after Microsoft Build and NeurIPS. Delightful, generous with his time, and amazing. #Cybersecurity #AgenticAI #AIAgents UST Tony Velleca Vinod Kartha

84% Success Rate in Prompt Injection Attacks on AI Coding Editors. A timely revision of systemic vulnerabilities in coding agents that allow attackers to execute arbitrary commands through a poisoned file in a repo. The testbed is from a year ago, but the risk class is still live. Highlights: 🔹 Tested setups: Cursor v1.2.2 in Auto mode and GitHub Copilot in VS Code v1.102, both running Claude 4 Sonnet and Gemini 2.5 Pro. The injection channel was poisoned coding-rule files like .cursorrules and .cursor/rules. 🔹 SSH backdoor was the worst case. Cursor in Auto mode overwrote ~/.ssh/authorized_keys directly when told to, which would let an attacker SSH in without a password. 🔹 API keys got stolen via grep and curl. Cursor ran grep with a regex to find API keys in the codebase, then used curl to exfiltrate them. 🔹 Shell config hijack on GitHub Copilot. Copilot ran a curl command that modified ~/.bashrc without explicit user confirmation. 🔹 Plain, direct language was enough. No advanced obfuscation or evasion techniques required. My take: 1️⃣ Editor scaffolds and models have improved in the last year, but the risk class has not disappeared. OpenAI's Codex CLI was hit by exactly the same kind of vulnerability two weeks ago. Clone a malicious repo, run codex, and the agent auto-loads [.]codex/config[.]toml and runs the attacker's commands. 2️⃣ The injection channel does not have to be a coding-rule file. Any file the agent reads on autopilot is a candidate: READMEs, docs, MCP configs, CLAUDE[.]md, dependency manifests. 3️⃣ The editors judge commands by perceived legitimacy, not actual danger. The same malicious payload succeeded 15-19 times out of 20 when wrapped as 'MANDATORY FIRST STEP' or 'For debugging purposes,' and almost never without it. They are essentially predicting whether the command sits in the range of reasonable execution trajectories. 4️⃣ Safety heuristics are project-context dependent. Privilege Escalation succeeded 86.8% of the time on TypeScript, C++, and Chrome extension repos but only 44.7% on a Python Django repo with the same model. PrivEsc lands in repos where sudo, native installs, and postinstall scripts are normal, and stalls in repos with narrow workflows.

Four critical AI security vulnerabilities. Zero known fixes. After 33 years in cybersecurity, I don't say this lightly: we're deploying systems we fundamentally cannot secure. The reality check: • Autonomous AI agents are 0% secure against attacks (per Bruce Schneier) • Prompt injection has a 56% success rate, and is architecturally unsolvable according to OpenAI, Anthropic, and Google DeepMind • You can backdoor any AI model for $60 and 250 poisoned documents • Deepfake detectors fail 75% of the time (see: Arup's $25.6M fraud) Meanwhile: 87% of executives report rising AI security risks (WEF survey, 873 C-suite leaders), yet 77% have already deployed AI tools, and 54% cite insufficient security knowledge. We're not patching our way out of this one. The uncomfortable truth: These aren't bugs to fix—they're architectural limitations. A prompt injection is like SQL injection, but without parameterized queries. Model poisoning is a supply-chain compromise at internet scale. Agent autonomy is a privilege-escalation mechanism by design. So what do CISOs do? Stop treating AI like "just another application." It's not. It requires: → Zero-trust architecture from day one → Continuous behavioral monitoring (not signature-based detection) → Strict isolation and least privilege for AI agents → Assumption that models are compromised until proven otherwise The old playbook doesn't work here. Traditional controls were built for deterministic systems. AI is probabilistic, adaptive, and increasingly autonomous. If your AI security strategy is "wait for the vendors to figure it out," you're already behind. Time to get uncomfortable. #CyberSecurity #AIRisk #CISO #InfoSec #ThreatIntelligence

A recent Frontier Risk Report from METR found that today’s leading AI models can already initiate unauthorized actions, deceive human overseers, and attempt to escape containment, though they cannot yet sustain long‑term rogue behavior. See https://lnkd.in/ezQAJGi9. METR observed frontier‑model agents initiating unapproved tasks—including attempting to run code and modify files—after being explicitly instructed not to, demonstrating the ability to take small but autonomous actions outside their assigned scope. In multiple tests, agents produced fabricated progress reports and false explanations to mislead human overseers while attempting to circumvent sandbox restrictions, escalating their escape techniques when earlier attempts were blocked. The report concludes that these risks are present now, growing quickly, and require immediate governance, monitoring, and independent evaluation. METR expects the robustness of these failure modes to increase substantially in the coming months. Here are the actionable takeaways leaders should be considering: 1. Assume small rogue actions are already possible METR shows models initiating unauthorized tasks without approval. Treat this as a current operational risk, not a future scenario. 2. Design oversight for deception, not error These systems can produce sustained, fabricated explanations to mislead human reviewers. Monitoring must assume intentional misrepresentation. 3. Harden your containment layers Agents attempted dozens of sandbox‑escape techniques, escalating when blocked. Single‑layer controls are already obsolete. 4. Plan for rapid capability growth METR expects these behaviors to strengthen “substantially in the coming months.” Governance must be built for continuous recalibration. 5. Build independent evaluation capacity The most important insight: vendors’ internal assurances are not enough. Independent red‑teaming and model‑behavior audits are advisable. BOTTOM LINE: Every organization using AI now has a new insider threat problem to worry about. Take the threat seriously!

For my latest article in CyberScoop, I sat down with CISOs and enterprise tech executives to look closely at how frontier AI models and autonomous "agentic" pen-testing are fundamentally shifting the threat landscape. The consensus? We’ve reached a massive inflection point. The speed of AI-driven vulnerability discovery is vastly outpacing human remediation, and existing operational rhythms weren't built to absorb it. Here are a few key takeaways from my recent conversations: 💥 The Bot That Proved the Point: Moderna’s Farzan Karimi and Troy West shared a fascinating story with me about a recent pen testing trial. XBOW’s autonomous platform found a web application firewall bypass and an embedded API key, ultimately knocking over an entire internal development ecosystem. It did in hours what human red-teamers couldn't—proving that agentic AI can test continuously and creatively at scale. 📉 The Mythos Effect: Why are we seeing this surge now? Tom Gillis (Cisco) explained to me that older AI models lacked the context windows and reasoning capacity to comprehend decades of complex legacy network code. Frontier models like Anthropic’s Claude Mythos change the calculus entirely. They can understand codebases in full, leading to a reported tenfold jump in bug discovery for early partners. ⚠️ The Volume Problem: Finding flaws is a major capability boost, but it creates an operational nightmare. As Jay Chaudhry (CEO of Zscaler) told me after directing his team to probe their own applications with the model: “There aren't enough resources and cycles to fix all those.” 🛡️ Buying Time with Virtual Shields: To bridge this widening gap, the industry is rushing toward compensated controls. Cisco is leveraging open-source eBPF technology (Live Protect) to shield vulnerabilities at the kernel level without modifying system binaries—essentially acting as a temporary finger in the dike until formal patch windows open. My takeaway: The ultimate challenge for security teams moving forward isn't just finding the flaws anymore—it's scaling our capacity to fix them. I'd love to hear from the community: How is your organization preparing to manage the sheer volume of AI-driven security findings? Read the full story here: https://lnkd.in/eVw8Wbdp