CTIBench: Evaluating LLMs in Cyber Threat Intelligence with Nidhi Rastogi - #729

CTIBench: Evaluating LLMs in Cyber Threat Intelligence with Nidhi Rastogi - #729

Today, we're joined by Nidhi Rastogi, assistant professor at Rochester Institute of Technology to discuss Cyber Threat Intelligence (CTI), focusing on her recent project CTIBench—a benchmark for evaluating LLMs on real-world CTI tasks. Nidhi explains the evolution of AI in cybersecurity, from rule-based systems to LLMs that accelerate analysis by providing critical context for threat detection and defense. We dig into the advantages and challenges of using LLMs in CTI, how techniques like Retrieval-Augmented Generation (RAG) are essential for keeping LLMs up-to-date with emerging threats, and how CTIBench measures LLMs’ ability to perform a set of real-world tasks of the cybersecurity analyst. We unpack the process of building the benchmark, the tasks it covers, and key findings from benchmarking various LLMs. Finally, Nidhi shares the importance of benchmarks in exposing model limitations and blind spots, the challenges of large-scale benchmarking, and the future directions of her AI4Sec Research Lab, including developing reliable mitigation techniques, monitoring "concept drift" in threat detection models, improving explainability in cybersecurity, and more. The complete show notes for this episode can be found at https://twimlai.com/go/729.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(794)

Do AI Tokenomics Matter More Than Model Benchmarks? with Chris Potts - #776

Do AI Tokenomics Matter More Than Model Benchmarks? with Chris Potts - #776

As reasoning models consume more tokens and AI systems become more expensive to run, understanding what those tokens actually buy is becoming increasingly important. In this episode, Stanford professo...

9 Sep 59min

World Models and the Future of Spatial AI with Justin Johnson - #775

World Models and the Future of Spatial AI with Justin Johnson - #775

In this episode, Justin Johnson, co-founder of World Labs, joins us to discuss world models and the emerging field of spatial AI. We explore why many researchers see capabilities beyond language as an...

1 Sep 1h 6min

Why the Next AI Breakthrough May Come from Physics with Max Welling - #774

Why the Next AI Breakthrough May Come from Physics with Max Welling - #774

The conventional wisdom in AI is that the next breakthrough will come from more compute, more data, and larger models. But what if the next leap comes from somewhere else? In this episode, Max Wellin...

26 Aug 58min

Why Image Generation Needs More Than Bigger Models with Fatih Porikli - #773

Why Image Generation Needs More Than Bigger Models with Fatih Porikli - #773

Text-to-image models have become remarkably good at producing realistic images. But realism isn’t the same as correctness. Ask for several distinct people, a specific composition, or a high-resolution...

12 Aug 56min

Why Models Are AI’s Next Training Dataset with Damian Borth - #772

Why Models Are AI’s Next Training Dataset with Damian Borth - #772

For more than a decade, AI has advanced by training ever-larger models on ever-larger datasets. But as high-quality training data becomes harder to find and pretraining grows increasingly expensive, r...

27 Juli 47min

How AI Learns to Smell with Alex Wiltschko - #771

How AI Learns to Smell with Alex Wiltschko - #771

In this episode, Alex Wiltschko, founder and CEO of Osmo, joins the show to discuss his goal of giving computers a sense of smell and what it takes to build olfactory intelligence. We explore the sci...

8 Juli 59min

Why AI Agents Break the GenAI Security Model with Devvret Rishi - #770

Why AI Agents Break the GenAI Security Model with Devvret Rishi - #770

In this episode, Sam talks with Dev Rishi, GM of AI at Rubrik, about what happens when agents move beyond answering questions and start taking action across tools, systems, and business processes. We...

16 Juni 56min

Is RAG Dead? Lessons from Building AI for Tax Law with Alex Bowcut - #769

Is RAG Dead? Lessons from Building AI for Tax Law with Alex Bowcut - #769

As context windows grow into the millions of tokens, many AI practitioners are questioning whether retrieval-augmented generation (RAG) is still necessary. If modern models can ingest entire libraries...

9 Juni 51min

Populärt inom Politik & nyheter

aftonbladet-krim
svenska-fall
p3-krim
fordomspodden
rss-krimstad
politiken
en-runda-till
flashback-forever
aftonbladet-daily
rss-vad-fan-hande
rss-krimreportrarna
rss-sanning-konsekvens
rss-flodet
svd-ledarredaktionen
motiv
rss-frandfors-horna
omni-podd
rss-expressen-dok
rss-utopia-2
spar