SIEM vs. Data Lake: Why We Ditched Traditional Logging?

SIEM vs. Data Lake: Why We Ditched Traditional Logging?

In this episode, Cliff Crosland, CEO & co-founder of Scanner.dev, shares his candid journey of trying (and initially failing) to build an in-house security data lake to replace an expensive traditional SIEM.

Cliff explains the economic breaking point where scaling a SIEM became "more expensive than the entire budget for the engineering team". He details the technical challenges of moving terabytes of logs to S3 and the painful realization that querying them with Amazon Athena was slow and costly for security use cases .

This episode is a deep dive into the evolution of logging architecture, from SQL-based legacy tools to the modern "messy" data lake that embraces full-text search on unstructured data. We discuss the "data engineering lift" required to build your own, the promise (and limitations) of Amazon Security Lake, and how AI agents are starting to automate detection engineering and schema management.


Guest Socials -⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cliff's Linkedin

Podcast Twitter - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@CloudSecPod⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our other Cloud Security Social Channels:

-⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Podcast- Youtube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Newsletter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

If you are interested in AI Cybersecurity, you can check out our sister podcast -⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ AI Security Podcast⁠


Questions asked:

(00:00) Introduction(02:25) Who is Cliff Crosford?(03:00) Why Teams Are Switching from SIEMs to Data Lakes(06:00) The "Black Hole" of S3 Logs: Cliff's First Failed Data Lake(07:30) The Engineering Lift: Do You Need a Data Engineer to Build a Lake?(11:00) Why Amazon Athena Failed for Security Investigations(14:20) The Danger of Dropping Logs to Save Costs(17:00) Misconceptions About Building Your Own Data Lake(19:00) The Evolution of Logging: From SQL to Full-Text Search(21:30) Is Amazon Security Lake the Answer? (OCSF & Custom Logs)(24:40) The Nightmare of Log Normalization & Custom Schemas(28:00) Why Future Tools Must Embrace "Messy" Logs(29:55) How AI Agents Are Automating Detection Engineering(35:45) Using AI to Monitor Schema Changes at Scale(39:45) Build vs. Buy: Does Your Security Team Need Data Engineers?(43:15) Fun Questions: Physics Simulations & Pumpkin Pie

Episoder(344)

A TECHNICAL WOMEN DELIVERING NON-TECHNICAL TRAINING IN CYBERSECURITY | HOW TO BUILD CYBERSECURITY TRAINING PROGRAM- Fareedah Shaheed, Online Safety and Security Strategist, Sekuva

A TECHNICAL WOMEN DELIVERING NON-TECHNICAL TRAINING IN CYBERSECURITY | HOW TO BUILD CYBERSECURITY TRAINING PROGRAM- Fareedah Shaheed, Online Safety and Security Strategist, Sekuva

In this episode, we sit with Fareedah Shaheed, Online Safety and Security Strategist @Sekuva. Fareedah & Ashish spoke about Importance of being technical for women in cybersecurity? Challenges of ...

29 Mar 202031min

How HASHICORP works with 90 percent Staff works Remotely | Incident Response | AWS Cloud Native! - Will Bengtson

How HASHICORP works with 90 percent Staff works Remotely | Incident Response | AWS Cloud Native! - Will Bengtson

In this episode, we sit with Will Bengtson, Director for Threat Detection and Response, Hashicorp. Will & Ashish spoke about What is Cloud Native & Cloud Security? How do you start with Threat Dete...

22 Mar 202057min

Multi Cloud Strategy | Multi Cloud Management for companies of all size - David Linthicum , Chief Cloud Strategy Officer for Delloite

Multi Cloud Strategy | Multi Cloud Management for companies of all size - David Linthicum , Chief Cloud Strategy Officer for Delloite

In this episode, we sit with David Linthicum, Chief Cloud Strategy Officer for Delloite. David & Ashish spoke about What is Cloud Security? How is security of data different/same in cloud from on-p...

15 Mar 202044min

CCPA COMPLIANCE | CALIFORNIA CONSUMER PRIVACY ACT | DATA GOVERNANCE BEST PRACTICES - TAYLOR HERSOM, VCISO, AUSTIN,TEXAS

CCPA COMPLIANCE | CALIFORNIA CONSUMER PRIVACY ACT | DATA GOVERNANCE BEST PRACTICES - TAYLOR HERSOM, VCISO, AUSTIN,TEXAS

In this episode, we sit with Taylor Hersom, vCISO, Austin,Texas. Taylor & Ashish spoke about Data privacy and Cloud Security California Consumer Privacy Act and how it affects all organisations aro...

8 Mar 202035min

Docker Security Best practice | Container Security 101 in AWS - Michael Hausenblas, Product Developer Advocate, AWS

Docker Security Best practice | Container Security 101 in AWS - Michael Hausenblas, Product Developer Advocate, AWS

Michael Hausenblas is a Product Developer Advocate, Amazon Web Services (AWS) Container Service team. Michael & Ashish spoke about Basics of Container Security Keeping Containers stateless vs buil...

1 Mar 202042min

Threat Intelligence platform for cyber security in Azure | Incident Response in Azure - Ashwin Patil, Threat Intelligence Center, Microsoft

Threat Intelligence platform for cyber security in Azure | Incident Response in Azure - Ashwin Patil, Threat Intelligence Center, Microsoft

Ashwin Patel is a Senior Program Manager, Threat Intelligence Microsoft. Ashwin & Ashish in this episode spoke about Capital One Data breach and how Azure Sentinel could have helped Setting up Secu...

1 Mar 202035min

MICROSOFT IGNITE 2020 SYDNEY | Getting started with securing Microsoft Azure Workload- David O'Brien, Microsoft MVP for Azure

MICROSOFT IGNITE 2020 SYDNEY | Getting started with securing Microsoft Azure Workload- David O'Brien, Microsoft MVP for Azure

David & Ashish spoke about What is Microsoft Ignite vs Microsoft Ignite Tour and why should cloud security people care about the event? Microsoft Ignite 2020 Sydney and Johannesburg What were th...

16 Feb 202046min

Is public cloud secure? - Francesco Cipollone, Cloud Security Alliance

Is public cloud secure? - Francesco Cipollone, Cloud Security Alliance

In this episode we speak to Francesco Cipollone, Head of Cloud Security Alliance for UK Francesco and Ashish speak about is public cloud secure and if multi-cloud is a good thing, especially if you a...

9 Feb 202034min

Populært innen Teknologi

lydartikler-fra-aftenposten
romkapsel
teknisk-sett
tomprat-med-gunnar-tjomlid
energi-og-klima
rss-impressions-2
shifter
fornybaren
nasjonal-sikkerhetsmyndighet-nsm
teknologi-og-mennesker
elektropodden
rss-ki-praten
rss-ai-forklart
smart-forklart
pedagogisk-intelligens
rss-for-alarmen-gar
rss-heis
rss-ki-til-kaffen
rss-digitaliseringspadden
rss-praktisk-proptech