Using the Smartest AI to Rate Other AI
Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(541)

UL NO. 416: Tracking AI Agent Activity, 400 SF Cameras, AI Sleeper Agents…

UL NO. 416: Tracking AI Agent Activity, 400 SF Cameras, AI Sleeper Agents…

Benign AI's "Many Eyes", OpenAI's Pentagon partnership, AI voice scams, Zuckerberg all-in on AGI, and more…Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy in...

24 Jan 202423min

A Conversation with Jason Kikta from Automox

A Conversation with Jason Kikta from Automox

In this sponsored episode of Unsupervised Learning, we talked to Jason Kikta. Jason is the CISO and Senior VP of Product at Automox, and our conversation covered: - Endpoint Management- IT and Securit...

22 Jan 202445min

UL NO. 415: It's Raining 9+ CVEs, 40% Job Loss from AI, Invisible Prompt Injection…

UL NO. 415: It's Raining 9+ CVEs, 40% Job Loss from AI, Invisible Prompt Injection…

[updated: apologies, we had the wrong audio file initially] Taiwan chooses democracy, 10,000 hours debunked, Data/Display/AI/AR, and much more… 📢Sponsored by Automox: Brace yourself for any IT calami...

19 Jan 202421min

UL NO. 414: LastPass Settings Upgrade, Boosting ChatGPT Output, AI Adding Societal Transparency

UL NO. 414: LastPass Settings Upgrade, Boosting ChatGPT Output, AI Adding Societal Transparency

ChatGPT prompting upgrades, CrewAI agent framework, people down on Democracy… 📢 Sponsored by Kolide: Concerned about data breaches and hacks? 🔒 Discover Kolide, the device trust solution that secure...

10 Jan 202425min

UL NO. 413: 7 Things to Expect from AI in 2024+, Xi Going Stalin, SSH's Terrapin…

UL NO. 413: 7 Things to Expect from AI in 2024+, Xi Going Stalin, SSH's Terrapin…

Xi purges detractors, my thoughts on chaos and 2024, my predictions for what we'll build with AI in 2024, macro D, and much more… Read online here.Become a Member: https://danielmiessler.com/upgradeSe...

6 Jan 202424min

A Conversation with Gabe Bernadett-Shapiro on AI

A Conversation with Gabe Bernadett-Shapiro on AI

👥 This conversation is between Daniel Miessler, founder of Unsupervised Learning, and Gabriel Bernadett-Shapiro, an expert on AI Safety and Threat Intelligence.  🧠 TOPICS 00:00:00 Intros 00:04:50 A...

21 Dec 202337min

UL NO. 412: OpenAI's Prompt Guide, My Neovim Overhaul, The UL Character Sheet, And…

UL NO. 412: OpenAI's Prompt Guide, My Neovim Overhaul, The UL Character Sheet, And…

Also: Ubiquity Cross-Pollination, Passcode Laws, China's AI Influence Network, Bodycam Shenanigans, And One Year Independent!Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/liste...

20 Dec 202329min

UL NO. 411: ChatGPT Repeat Vuln, A UL AI Course!, Revenge Code Deletion

UL NO. 411: ChatGPT Repeat Vuln, A UL AI Course!, Revenge Code Deletion

Sneaky ChatGPT Data Leaks, A New Ground-Based Telescope, Companies Leaving Austin, More… 📢Sponsored by Automox: Brace yourself for any IT calamity with Automox! 🛡️ Tune into the Autonomous IT podcas...

14 Dec 202319min

Populärt inom Teknik

uppgang-och-fall
elbilsveckan
market-makers
rss-laddstationen-med-elbilen-i-sverige
rss-technokratin
natets-morka-sida
rss-elektrikerpodden
skogsforum-podcast
rss-uppgang-och-fall
bilar-med-sladd
rss-powerboat-sverige-podcast
developers-mer-an-bara-kod
bli-saker-podden
rss-it-sakerhetspodden
rss-digitala-influencer-podden
rss-veckans-ai
hej-bruksbil
rss-fabriken-2
rss-en-ai-till-kaffet
rss-snacka-om-ai