Using the Smartest AI to Rate Other AI

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Avsnitt(532)

Unsupervised Learning: No. 52

Unsupervised Learning: No. 52

Subscribe to Unsupervised Learning via: iTunes | Android | RSS | Newsletter This is Episode No. 52 of Unsupervised Learning—a weekly show where I collect my favorite stories in infosec, technology, and humans, and talk about why they matter. The show is released in two forms: * the Podcast, which you can subscribe to via iTunes, Android, Blog, or RSS * the Newsletter, which is the podcast’s companion and conveniently serves as its show notes as well. You can get the newsletter by clicking here or on the image/text below. Thank you for listening! Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

7 Nov 201640min

Unsupervised Learning: No. 51

Unsupervised Learning: No. 51

Subscribe to Unsupervised Learning via: iTunes | Android | RSS | Newsletter This is Episode No. 51 of Unsupervised Learning—a weekly show where I collect my favorite stories in infosec, technology, and humans, and talk about why they matter. The show is released in two forms: * the Podcast, which you can subscribe to via iTunes, Android, Blog, or RSS * the Newsletter, which is the podcast’s companion and conveniently serves as its show notes as well. You can get the newsletter by clicking here or on the image/text below. Thank you for listening! Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

31 Okt 20161h

Unsupervised Learning: No. 50

Unsupervised Learning: No. 50

Subscribe to Unsupervised Learning via: iTunes | Android | RSS | Newsletter This is Episode No. 50 of Unsupervised Learning—a weekly show where I collect my favorite stories in infosec, technology, and humans, and talk about why they matter. The show is released in two forms: * the Podcast, which you can subscribe to via iTunes, Android, Blog, or RSS * the Newsletter, which is the podcast’s companion and conveniently serves as its show notes as well. You can get the newsletter by clicking here or on the image/text below. Thank you for listening! Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

24 Okt 201637min

Unsupervised Learning: No. 49

Unsupervised Learning: No. 49

Subscribe to Unsupervised Learning via: iTunes | Android | RSS | Newsletter This is Episode No. 49 of Unsupervised Learning—a weekly show where I collect my favorite stories in infosec, technology, and humans, and talk about why they matter. The show is released in two forms: * the Podcast, which you can subscribe to via iTunes, Android, Blog, or RSS * the Newsletter, which is the podcast’s companion and conveniently serves as its show notes as well. You can get the newsletter by clicking here or on the image/text below. Thank you for listening! Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

18 Okt 201647min

Unsupervised Learning: Episode 46

Unsupervised Learning: Episode 46

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter [ Click here to get the full companion newsletter with complete show notes from this episode. ] [ Click here to get the full companion newsletter with complete show notes from this episode. ] Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

27 Sep 201630min

Unsupervised Learning: Episode 45

Unsupervised Learning: Episode 45

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter [ Click here to get the full companion newsletter with complete show notes from this episode. ] [ Click here to get the full companion newsletter with complete show notes from this episode. ] Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

19 Sep 201657min

Unsupervised Learning: Episode 44

Unsupervised Learning: Episode 44

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter [ Click here to view the full companion newsletter with complete show notes from this episode. ] [ Click here to view the full companion newsletter with complete show notes from this episode. ] Subscribe to the Podcast via: iTunes | Android | RSS | NewsletterBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

11 Sep 201634min

Unsupervised Learning: Episode 43

Unsupervised Learning: Episode 43

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter News Internet disinformation service for hire [ Link ] Rob Fuller (@mubix) has found a way to pull credentials from a locked machine using a USB dongle and Responder [ Link ] Yelp starts new bug bounty with HackerOne, offers up to 15K […] -- :: Unsupervised Learning: Episode 43 appeared originally on danielmiessler.com. :: Subscribe to Unsupervised Learning---my weekly show where I handpick the best stories from infosec and technology, and talk about why they matter.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

7 Sep 201642min

Populärt inom Teknik

uppgang-och-fall
elbilsveckan
rss-racevecka
bilar-med-sladd
market-makers
skogsforum-podcast
rss-laddstationen-med-elbilen-i-sverige
rss-technokratin
natets-morka-sida
rss-elektrikerpodden
developers-mer-an-bara-kod
mediepodden
ai-sweden-podcast
rss-uppgang-och-fall
solcellskollens-podcast
hej-bruksbil
bli-saker-podden
rss-it-sakerhetspodden
rss-veckans-ai
rss-fabriken-2