SIEM vs. Data Lake: Why We Ditched Traditional Logging?

SIEM vs. Data Lake: Why We Ditched Traditional Logging?

In this episode, Cliff Crosland, CEO & co-founder of Scanner.dev, shares his candid journey of trying (and initially failing) to build an in-house security data lake to replace an expensive traditional SIEM.

Cliff explains the economic breaking point where scaling a SIEM became "more expensive than the entire budget for the engineering team". He details the technical challenges of moving terabytes of logs to S3 and the painful realization that querying them with Amazon Athena was slow and costly for security use cases .

This episode is a deep dive into the evolution of logging architecture, from SQL-based legacy tools to the modern "messy" data lake that embraces full-text search on unstructured data. We discuss the "data engineering lift" required to build your own, the promise (and limitations) of Amazon Security Lake, and how AI agents are starting to automate detection engineering and schema management.


Guest Socials -⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cliff's Linkedin

Podcast Twitter - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@CloudSecPod⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our other Cloud Security Social Channels:

-⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Podcast- Youtube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Newsletter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

If you are interested in AI Cybersecurity, you can check out our sister podcast -⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ AI Security Podcast⁠


Questions asked:

(00:00) Introduction(02:25) Who is Cliff Crosford?(03:00) Why Teams Are Switching from SIEMs to Data Lakes(06:00) The "Black Hole" of S3 Logs: Cliff's First Failed Data Lake(07:30) The Engineering Lift: Do You Need a Data Engineer to Build a Lake?(11:00) Why Amazon Athena Failed for Security Investigations(14:20) The Danger of Dropping Logs to Save Costs(17:00) Misconceptions About Building Your Own Data Lake(19:00) The Evolution of Logging: From SQL to Full-Text Search(21:30) Is Amazon Security Lake the Answer? (OCSF & Custom Logs)(24:40) The Nightmare of Log Normalization & Custom Schemas(28:00) Why Future Tools Must Embrace "Messy" Logs(29:55) How AI Agents Are Automating Detection Engineering(35:45) Using AI to Monitor Schema Changes at Scale(39:45) Build vs. Buy: Does Your Security Team Need Data Engineers?(43:15) Fun Questions: Physics Simulations & Pumpkin Pie

Jaksot(343)

How HASHICORP works with 90 percent Staff works Remotely | Incident Response | AWS Cloud Native! - Will Bengtson

How HASHICORP works with 90 percent Staff works Remotely | Incident Response | AWS Cloud Native! - Will Bengtson

In this episode, we sit with Will Bengtson, Director for Threat Detection and Response, Hashicorp. Will & Ashish spoke about What is Cloud Native & Cloud Security? How do you start with Threat Dete...

22 Maalis 202057min

Multi Cloud Strategy | Multi Cloud Management for companies of all size - David Linthicum , Chief Cloud Strategy Officer for Delloite

Multi Cloud Strategy | Multi Cloud Management for companies of all size - David Linthicum , Chief Cloud Strategy Officer for Delloite

In this episode, we sit with David Linthicum, Chief Cloud Strategy Officer for Delloite. David & Ashish spoke about What is Cloud Security? How is security of data different/same in cloud from on-p...

15 Maalis 202044min

CCPA COMPLIANCE | CALIFORNIA CONSUMER PRIVACY ACT | DATA GOVERNANCE BEST PRACTICES - TAYLOR HERSOM, VCISO, AUSTIN,TEXAS

CCPA COMPLIANCE | CALIFORNIA CONSUMER PRIVACY ACT | DATA GOVERNANCE BEST PRACTICES - TAYLOR HERSOM, VCISO, AUSTIN,TEXAS

In this episode, we sit with Taylor Hersom, vCISO, Austin,Texas. Taylor & Ashish spoke about Data privacy and Cloud Security California Consumer Privacy Act and how it affects all organisations aro...

8 Maalis 202035min

Docker Security Best practice | Container Security 101 in AWS - Michael Hausenblas, Product Developer Advocate, AWS

Docker Security Best practice | Container Security 101 in AWS - Michael Hausenblas, Product Developer Advocate, AWS

Michael Hausenblas is a Product Developer Advocate, Amazon Web Services (AWS) Container Service team. Michael & Ashish spoke about Basics of Container Security Keeping Containers stateless vs buil...

1 Maalis 202042min

Threat Intelligence platform for cyber security in Azure | Incident Response in Azure - Ashwin Patil, Threat Intelligence Center, Microsoft

Threat Intelligence platform for cyber security in Azure | Incident Response in Azure - Ashwin Patil, Threat Intelligence Center, Microsoft

Ashwin Patel is a Senior Program Manager, Threat Intelligence Microsoft. Ashwin & Ashish in this episode spoke about Capital One Data breach and how Azure Sentinel could have helped Setting up Secu...

1 Maalis 202035min

MICROSOFT IGNITE 2020 SYDNEY | Getting started with securing Microsoft Azure Workload- David O'Brien, Microsoft MVP for Azure

MICROSOFT IGNITE 2020 SYDNEY | Getting started with securing Microsoft Azure Workload- David O'Brien, Microsoft MVP for Azure

David & Ashish spoke about What is Microsoft Ignite vs Microsoft Ignite Tour and why should cloud security people care about the event? Microsoft Ignite 2020 Sydney and Johannesburg What were th...

16 Helmi 202046min

Is public cloud secure? - Francesco Cipollone, Cloud Security Alliance

Is public cloud secure? - Francesco Cipollone, Cloud Security Alliance

In this episode we speak to Francesco Cipollone, Head of Cloud Security Alliance for UK Francesco and Ashish speak about is public cloud secure and if multi-cloud is a good thing, especially if you a...

9 Helmi 202034min

Just Eat UK security - cloud security across Scotland uk canada in a world of multi public cloud

Just Eat UK security - cloud security across Scotland uk canada in a world of multi public cloud

In this episode we speak to Stu Hirst, Principal Cloud Security @Just Eat. Stu and Ashish speak about keeping up security in a world of multi cloud, the challenges of recruiting for cloud security, w...

2 Helmi 20201h 4min