Using the Smartest AI to Rate Other AI

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Episoder(531)

Novelty Exploration vs. Pattern Exploitation

Novelty Exploration vs. Pattern Exploitation

How going from exploration to exploitation can help you as both a consumer and creator of everything.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

15 Okt 3min

Magnifying Time

Magnifying Time

Some thoughts on how novelty and attention magnify the time that we have. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

14 Okt 6min

A Conversation With Harry Wetherald CO-Founder & CEO At Maze

A Conversation With Harry Wetherald CO-Founder & CEO At Maze

➡ Stay Ahead of Cyber Threats with AI-Driven Vulnerability Management with Maze:https://mazehq.com/ In this conversation, I speak with Harry about how AI is transforming vulnerability management and application security. We explore how modern approaches can move beyond endless reports and generic fixes, toward real context-aware workflows that actually empower developers and security teams. We talk about: The Real Problem in Vulnerability ManagementWhy remediation—not just prioritization—remains the toughest challenge, and how AI can help bridge the gap between vulnerabilities and the developers who need to fix them. Context, Ownership, and VelocityHow linking vulnerabilities to the right applications and teams inside their daily tools (like GitHub) reduces friction, speeds up patching, and improves security without slowing developers down. AI Agents and the Future of SecurityWhy we should think of AI agents as “extra eyes and hands,” and how they’re reshaping everything from threat detection to system design, phishing campaigns, and organizational defense models. Attackers Move FirstHow attackers are already building unified world models of their targets using AI, and why defenders need to match (or exceed) this intelligence to stay ahead. From Days to MinutesWhy the tolerance for vulnerability windows is shrinking fast, and how automation and AI are pushing us toward a future where hours—or even minutes—make the difference. Subscribe to the newsletter at:https://danielmiessler.com/subscribe Join the UL community at:https://danielmiessler.com/upgrade Follow on X:https://x.com/danielmiessler Follow on LinkedIn:https://www.linkedin.com/in/danielmiessler Chapters: 00:00 – Welcome and Harry’s Background01:07 – The Real Problem: Remediation vs. Prioritization04:31 – Breaking Down Vulnerability Context and Threat Intel05:46 – Connecting Vulnerabilities to Developers and Workflows08:01 – Why Traditional Vulnerability Management Fails10:29 – Startup Lessons and The State of AI Agents13:26 – DARPA’s AI Cybersecurity Competition14:29 – System Design: Deterministic Code vs. AI16:05 – How the Product Works and Data Sources18:01 – AI as “Extra Eyes and Hands” in Security20:20 – Breaking Barriers: Rethinking Scale with AI23:22 – Building World Models for Defense (and Attack)25:22 – Attackers Move Faster: Why Context Matters27:04 – Phishing at Scale with AI Agents31:24 – Shrinking Windows of Vulnerability: From Days to Minutes32:47 – What’s Next for Harry’s Work34:13 – Closing ThoughtsBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

22 Sep 35min

A Conversation With Grant Lee CO-Founder & CEO At Gamma

A Conversation With Grant Lee CO-Founder & CEO At Gamma

➡ Upgrade your presentations with Gamma, the best AI presentation maker: https://gamma.app In this conversation, I speak with Grant, co-founder of Gamma, about how their platform is transforming presentations and idea-sharing. Instead of starting with slides, Gamma helps you focus on the story first—then builds the visuals, structure, and delivery around it using AI. We talk about: From Slides to StoriesWhy presentations should begin with narrative flow and core ideas, not pre-existing slide templates. Gamma enables creators to design around the message rather than being trapped by the format. AI as Your Presentation PartnerHow Gamma acts like a personal design expert—adjusting layouts, visuals, and style in real time—similar to having a world-class presentation coach and designer by your side. Idea Propagation Beyond SlidesWhy Gamma isn’t just about “slides,” but about propagating ideas in the right medium: presentations, video overlays, mobile-first content, or even context-based imagery and clips. The Future of GammaWhere the platform is headed in the next few years, and how AI-driven storytelling will redefine the way we share ideas across industries. Subscribe to the newsletter:https://danielmiessler.com/subscribe Join the UL community:https://danielmiessler.com/upgrade Follow on X:https://x.com/danielmiessler Follow on LinkedIn:https://www.linkedin.com/in/danielmiessler Chapters: 00:00 – Introduction to Unsupervised Learning00:17 – Welcome Grant and Gamma’s Background01:31 – AI Trends Driving Presentation Innovation03:20 – Story First: Rethinking Workflow Beyond Slides04:29 – Building Narrative Flow Before Design07:42 – Gamma as an AI Presentation Partner09:43 – What Gamma Does Differently from Other Tools12:27 – Idea Propagation: Matching Message, Medium, and Audience13:23 – Enhancing Presentations with Images, Clips, and Context15:15 – Current Graphics and Animation Options17:03 – Most Popular and Favorite Features in Gamma18:05 – What’s Coming Soon in Gamma19:08 – The Future of Idea Propagation with AI20:46 – Where to Learn More About Gamma21:21 – Closing ThoughtsBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

18 Sep 21min

UL NO. 497: STANDARD EDITION | More NPM Shenanigans, I Open Sourced Kai, Blood Work Results, Finding Vulns in a 10-line Prompt, and more...

UL NO. 497: STANDARD EDITION | More NPM Shenanigans, I Open Sourced Kai, Blood Work Results, Finding Vulns in a 10-line Prompt, and more...

UL NO. 497: STANDARD EDITION | More NPM Shenanigans, I Open Sourced Kai, Blood Work Results, Finding Vulns in a 10-line Prompt, and more... Read this episode online: https://newsletter.danielmiessler.com/p/ul-497 Subscribe to the newsletter at:https://danielmiessler.com/subscribe Join the UL community at:https://danielmiessler.com/upgrade Follow on X:https://x.com/danielmiessler Follow on LinkedIn:https://www.linkedin.com/in/danielmiesslerBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

10 Sep 37min

UL NO. 496: STANDARD EDITION | New Video on Building my Personal AI System, Anthropic Reveals One-person Hacking Company using Claude, Pentagon Says China Keeps Penetrating, and more...

UL NO. 496: STANDARD EDITION | New Video on Building my Personal AI System, Anthropic Reveals One-person Hacking Company using Claude, Pentagon Says China Keeps Penetrating, and more...

UL NO. 496: STANDARD EDITION | New Video on Building my Personal AI System, Anthropic Reveals One-person Hacking Company using Claude, Pentagon Says China Keeps Penetrating, and more... Read this episode online: https://newsletter.danielmiessler.com/p/ul-496 Personal AI Video I'm so excited about Subscribe to the newsletter at:https://danielmiessler.com/subscribe Join the UL community at:https://danielmiessler.com/upgrade Follow on X:https://x.com/danielmiessler Follow on LinkedIn:https://www.linkedin.com/in/danielmiesslerBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

5 Sep 1h 2min

A Conversation with Michael Brown About Designing AI Systems

A Conversation with Michael Brown About Designing AI Systems

In this episode of Unsupervised Learning, I sit down with Michael Brown, Principal Security Engineer at Trail of Bits, to dive deep into the design and lessons learned from the AI Cyber Challenge (AIxCC). Michael led the team behind Buttercup, an AI-driven system that secured 2nd place overall. We discuss: -The design philosophy behind Buttercup and how it blended deterministic systems with AI/ML -Why modular architectures and “best of both worlds” approaches outperform pure LLM-heavy -designs -How large language models performed in patch generation and fuzzing support -The risks of compounding errors in AI pipelines — and how to avoid them -Broader lessons for applying AI in cybersecurity and beyond If you’re interested in AI, security engineering, or system design at scale, this conversation breaks down what worked, what didn’t, and where the field is heading. Subscribe to the newsletter at:https://danielmiessler.com/subscribe Join the UL community at:https://danielmiessler.com/upgrade Follow on X:https://x.com/danielmiessler Follow on LinkedIn:https://www.linkedin.com/in/danielmiesslerBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

22 Aug 50min

UL NO. 494:  STANDARD EDITION | AI Finds a P1, I Missed Chartbeat So I Made My Own, XBow Open-Sources Their AI Bot, and more...

UL NO. 494:  STANDARD EDITION | AI Finds a P1, I Missed Chartbeat So I Made My Own, XBow Open-Sources Their AI Bot, and more...

You are currently listening to the Standard version of the podcast, consider upgrading and becoming a member to unlock the full version and many other exclusive benefits here: https://newsletter.danielmiessler.com/upgrade Read this episode online: https://newsletter.danielmiessler.com/p/ul-494 Subscribe to the newsletter at:https://danielmiessler.com/subscribe Join the UL community at:https://danielmiessler.com/upgrade Follow on X:https://x.com/danielmiessler Follow on LinkedIn:https://www.linkedin.com/in/danielmiesslerBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

21 Aug 1h 38min

Populært innen Teknologi

romkapsel
rss-avskiltet
teknisk-sett
tomprat-med-gunnar-tjomlid
energi-og-klima
rss-impressions-2
nasjonal-sikkerhetsmyndighet-nsm
shifter
elektropodden
fornybaren
rss-alt-vi-kan
smart-forklart
rss-snakk-om-sikkerhet
rss-bouvet-bobler
teknologi-og-mennesker
pedagogisk-intelligens
kunstig-intelligens-med-morten-goodwin
rss-digitaliseringspadden
rss-alt-som-gar-pa-strom
rss-hvorfor-ble-det-sann