Using the Smartest AI to Rate Other AI
Unsupervised Learning19 Huhti 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(541)

UL NO. 466 | My Analysis and Prediction on the Deepseek Situation

UL NO. 466 | My Analysis and Prediction on the Deepseek Situation

Plus: The AI Vulnerability Glut, Remotely Hacking Subarus, Criticism of CVSS, the United Breach, and much more... ➡ Protect Against Bots, Fraud, and Abuse. Check out WorkOS Radar at workos.com/radar S...

30 Tammi 202533min

A Conversation with Faisal Khan from Vanta

A Conversation with Faisal Khan from Vanta

In this episode, I speak with Faisal Khan, a GRC Solution Specialist at Vanta, about how their platform is transforming trust management for organizations. We talk about: Vanta as a Trust-Management P...

28 Tammi 202539min

UL NO. 465 | The SaaS Attack Vector, Project Stargate, and Undersea Cable Drones

UL NO. 465 | The SaaS Attack Vector, Project Stargate, and Undersea Cable Drones

also...Joseph goes independent, Perplexity's new search API, Stoicism's gift, and much more... Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at:https://dan...

26 Tammi 202521min

UL NO. 464 | AI Phishing Matches Humans, Under Sea Cable Cutter Patents, and Siri is About to Not Suck

UL NO. 464 | AI Phishing Matches Humans, Under Sea Cable Cutter Patents, and Siri is About to Not Suck

also...Russia's actual playbook, CISA's new rating system, and everyone's doing robots now Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at:https://danielm...

18 Tammi 202528min

UL NO. 463 | Launching 2025, US Soldier Data Leak, AI Agents Emerge, China's Global Spy Network, Robotaxis Now Safer Than Humans

UL NO. 463 | Launching 2025, US Soldier Data Leak, AI Agents Emerge, China's Global Spy Network, Robotaxis Now Safer Than Humans

Navigating AI's impact on work, the rise of transnational threats, a grim new reality in air travel, and how to harness the chaos of 2025 for personal and professional growth. Subscribe to the newslet...

11 Tammi 202544min

UL NO. 462: Full-Face Mask Deceptions, VS Code Tunnel Hacks, Quiet AI Emergence at Apple, and Tokyo’s Three-Day Weekend Gamble

UL NO. 462: Full-Face Mask Deceptions, VS Code Tunnel Hacks, Quiet AI Emergence at Apple, and Tokyo’s Three-Day Weekend Gamble

...plus building personal TELOS files, the ChatGPT Pro vs. Claude coding face-off, a human bird flu case in Louisiana, and ketones fighting Alzheimer’s. ➡ Make your app enterprise-ready and start sel...

22 Joulu 202427min

How Much AI Do We Need? - My AI Industry Prediction

How Much AI Do We Need? - My AI Industry Prediction

In this episode, Daniel Miessler explores how AI can transform our understanding of the present and create actionable paths for a better future. He talks about: The Current State, Desired State, and T...

11 Joulu 202428min

UL NO. 459: New Active 0-day Exploitation, AI That Sees Your Open Apps, The RebootAI Project

UL NO. 459: New Active 0-day Exploitation, AI That Sees Your Open Apps, The RebootAI Project

A conversation with Rob Allen from ThreatLocker, UL's Black Friday sale, Finland's internet disrupted, and more... ➡️ Get Your Free Cloud Security Scan with Wiz: wiz.io/ul Subscribe to the newsletter ...

21 Marras 202423min