Building a Security Agent You Can Actually Trust

​Off the Record - a Noise2Signal Session Engineering Series: Your Agent Is a Distributed System, Act Like It AI agents are quickly moving from demos into real security workflows. But there’s a big difference between an agent that can call a tool and one you’d actually trust to investigate a vulnerability, analyze evidence, or take action in a production environment. Join a small group of security engineers, architects, builders, and practitioners for a technical evening focused on what it actually takes to build reliable AI agents for security work. We’ll walk through lessons from building production security agents at Quantro Security, including what worked, what failed, and where we learned not to trust the model. No vendor presentation. No generic “AI is changing cybersecurity” talk. We’re going to build, break things down, and compare notes. What we'll cover From chatbot to agent: When a security problem actually benefits from an agent versus a deterministic workflow or simple LLM interaction. Giving agents real tools: APIs, scanners, asset data, code repositories, shell access, and other systems, and how to constrain what the agent can do. From answers to evidence: Designing agents that have to prove their conclusions instead of generating convincing-sounding security analysis. What happens when the data disagrees: Stale scanner results, conflicting signals, missing evidence, and knowing when the agent should say I don't know. Evals and verification: How do you know your security agent actually completed the task correctly? Autonomy boundaries: What should an agent be allowed to decide, what should remain deterministic, and where humans stay in the loop. Lessons from production: Failure modes, architecture decisions, and things that looked great in a prototype but didn't survive contact with the real world. The walkthrough We'll use a vulnerability investigation as the working example: Given a vulnerability and an asset, can an agent determine whether the issue is actually exploitable, and produce enough evidence for a security engineer to trust the result? We'll start with the naive version, progressively give it tools and context, break it a few times, and work toward an architecture that can make a defensible conclusion. The concepts apply well beyond vulnerability management; detection triage, cloud security, AppSec, IAM, incident investigation, remediation, and other security workflows. Agenda 6:00 — Pizza, drinks & introductionsMeet other NYC security practitioners and builders. 6:30 — Talk BeginsTechnical walkthrough: designing and building a reliable security agent. 7:15 — Open discussion & networkingCompare approaches, architectures, and ideas over pizza and drinks. 8:00 — Wrap-up Who should come Security engineers, security architects, vulnerability management and exposure management practitioners, AppSec engineers, security automation/platform engineers, detection engineers, engineering-minded security leaders, and anyone actively experimenting with AI agents for security. You don't need to already be building agents. You should just be interested in understanding how to make them useful and trustworthy for real security work. House rules This is a practitioner event, not a sales event. Ask technical questions. Challenge the architecture. Share what you've tried. Talk about what failed. No vendor pitches. No recruiting pitches. Just security practitioners comparing notes and building things. Attendance is intentionally limited so the session stays interactive. Your host: Sasan Padidar — Co-founder & CEO, Quantro Security Hosted by the Noise2Signal team. Sponsored by Quantro Security. Noise2Signal is a cybersecurity podcast - candid, vendor-pitch-free conversations with the people who built the industry. This dinner runs on the same rules. Quantro Security picks up the check: an autonomous AI agent platform built to give defenders the tooling and automation attackers already have. That's the whole pitch.