Using the Smartest AI to Rate Other AI

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Episoder(531)

Unsupervised Learning: No. 51

Unsupervised Learning: No. 51

Subscribe to Unsupervised Learning via: iTunes | Android | RSS | Newsletter This is Episode No. 51 of Unsupervised Learning—a weekly show where I collect my favorite stories in infosec, technology, and humans, and talk about why they matter. The show is released in two forms: * the Podcast, which you can subscribe to via iTunes, Android, Blog, or RSS * the Newsletter, which is the podcast’s companion and conveniently serves as its show notes as well. You can get the newsletter by clicking here or on the image/text below. Thank you for listening! Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

31 Okt 20161h

Unsupervised Learning: No. 50

Unsupervised Learning: No. 50

Subscribe to Unsupervised Learning via: iTunes | Android | RSS | Newsletter This is Episode No. 50 of Unsupervised Learning—a weekly show where I collect my favorite stories in infosec, technology, and humans, and talk about why they matter. The show is released in two forms: * the Podcast, which you can subscribe to via iTunes, Android, Blog, or RSS * the Newsletter, which is the podcast’s companion and conveniently serves as its show notes as well. You can get the newsletter by clicking here or on the image/text below. Thank you for listening! Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

24 Okt 201637min

Unsupervised Learning: No. 49

Unsupervised Learning: No. 49

Subscribe to Unsupervised Learning via: iTunes | Android | RSS | Newsletter This is Episode No. 49 of Unsupervised Learning—a weekly show where I collect my favorite stories in infosec, technology, and humans, and talk about why they matter. The show is released in two forms: * the Podcast, which you can subscribe to via iTunes, Android, Blog, or RSS * the Newsletter, which is the podcast’s companion and conveniently serves as its show notes as well. You can get the newsletter by clicking here or on the image/text below. Thank you for listening! Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

18 Okt 201647min

Unsupervised Learning: Episode 46

Unsupervised Learning: Episode 46

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter [ Click here to get the full companion newsletter with complete show notes from this episode. ] [ Click here to get the full companion newsletter with complete show notes from this episode. ] Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

27 Sep 201630min

Unsupervised Learning: Episode 45

Unsupervised Learning: Episode 45

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter [ Click here to get the full companion newsletter with complete show notes from this episode. ] [ Click here to get the full companion newsletter with complete show notes from this episode. ] Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter Notes * The intro track is from one of my favorite EDM artists: Zomby. The song is ‘Orion’, and it’s from the ‘With Love’ album. Highly recommended if you like chill EDM. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

19 Sep 201657min

Unsupervised Learning: Episode 44

Unsupervised Learning: Episode 44

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter [ Click here to view the full companion newsletter with complete show notes from this episode. ] [ Click here to view the full companion newsletter with complete show notes from this episode. ] Subscribe to the Podcast via: iTunes | Android | RSS | NewsletterBecome a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

11 Sep 201634min

Unsupervised Learning: Episode 43

Unsupervised Learning: Episode 43

Subscribe to the Podcast via: iTunes | Android | RSS | Newsletter News Internet disinformation service for hire [ Link ] Rob Fuller (@mubix) has found a way to pull credentials from a locked machine using a USB dongle and Responder [ Link ] Yelp starts new bug bounty with HackerOne, offers up to 15K […] -- :: Unsupervised Learning: Episode 43 appeared originally on danielmiessler.com. :: Subscribe to Unsupervised Learning---my weekly show where I handpick the best stories from infosec and technology, and talk about why they matter.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

7 Sep 201642min

Unsupervised Learning: Episode 42

Unsupervised Learning: Episode 42

[ Subscribe to the Podcast: iTunes | Android | RSS ] InfoSec news and articles Dropbox hacked 68 million accounts Back in 2012 Malware infected all Eddie Bauer stores in U.S. and Canada All 350 stores in North America Wicked iPhone vulnerability called Trident (3 0days) All you need to do is follow a link, […] -- :: Unsupervised Learning: Episode 42 appeared originally on danielmiessler.com. :: Subscribe to Unsupervised Learning---my weekly show where I handpick the best stories from infosec and technology, and talk about why they matter.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

1 Sep 20161h 4min

Populært innen Teknologi

romkapsel
rss-avskiltet
teknisk-sett
tomprat-med-gunnar-tjomlid
energi-og-klima
rss-impressions-2
shifter
nasjonal-sikkerhetsmyndighet-nsm
elektropodden
fornybaren
rss-alt-vi-kan
rss-alt-som-gar-pa-strom
smart-forklart
rss-snakk-om-sikkerhet
teknologi-og-mennesker
kunstig-intelligens-med-morten-goodwin
rss-bouvet-bobler
i-loopen
pedagogisk-intelligens
rss-digitaliseringspadden