Using the Smartest AI to Rate Other AI

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Avsnitt(532)

News & Analysis | No. 285

News & Analysis | No. 285

The latest in Security News, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Aphorism… Web Version: https://danielmiessler.com/podcast/news-analysis-no-285/Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

14 Juni 20218min

News & Analysis | No. 284

News & Analysis | No. 284

The latest in Security News, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Aphorism… Web Version: https://danielmiessler.com/podcast/news-analysis-no-284/Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

7 Juni 202120min

News & Analysis | No. 283

News & Analysis | No. 283

Conti Ransomware Attacks Against US Targets, GPT-3 Disinformation Sways Opinion, SolarWinds Group Has New NativeZone Tool, Open Source HIBP, CASM, Autonomous Drone Attack, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Aphorism…Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

1 Juni 202126min

News & Analysis | No. 282

News & Analysis | No. 282

Pentagon Civilians and Contractors, CISA SolarWinds, CNA, DarkSide Money, China RSA, Senate Science Bill, Google RSS, Technology News, Human News, Notes, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Aphorism…Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

24 Maj 202125min

News & Analysis | No. 281

News & Analysis | No. 281

Darkside Colonial, Cyber Executive Order, DBIR 2021, WiFi Vulns, Microsoft AI Security, OpenSSH Hardware Keys, Insurer AXA Ransomed, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Aphorism…Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

18 Maj 202122min

News & Analysis | No. 280

News & Analysis | No. 280

Oil Pipeline Ransomware, NSA OT Warning, Deepfake Uptick, Insurer Stops Ransomware Payouts, Google Automatic 2FA, AI-powered Cameras in Banks, Technology News, Content, Ideas & Analysis, Notes, Discovery, Recommendations, and the Aphorism of the Week…Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

10 Maj 202124min

News & Analysis | No. 279

News & Analysis | No. 279

FBI and CISA release SVR (Cozy Bear) TTPs, CISA releases an RTOS advisory around ICS, a task force has a plan for the Biden administration to counter ransomware, there's a vulnerability in the ipaddress library in Python, Krebs says Experian leaked credit scores, Censys found 1.93 million online databases, Technology News, Content, Ideas & Analysis, Notes, Discovery, Recommendations, and the Aphorism of the Week…Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

3 Maj 202121min

News & Analysis | No. 278

News & Analysis | No. 278

New Logo, CISA Pulse Secure VPN Orders, Ransomware Gangs + Stock Shorter Scams, Japanese NASA Hacked by China, Dan Kaminsky, LinkedIn Brittons and China/Russia, Codecov Fallout, Technology News, Content, Ideas & Analysis, Notes, Discovery, Recommendations, and the Aphorism of the Week…Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

26 Apr 202112min

Populärt inom Teknik

uppgang-och-fall
rss-racevecka
elbilsveckan
bilar-med-sladd
market-makers
rss-badfluence
skogsforum-podcast
rss-uppgang-och-fall
rss-technokratin
natets-morka-sida
rss-elektrikerpodden
developers-mer-an-bara-kod
hej-bruksbil
rss-digitala-influencer-podden
rss-veckans-ai
har-vi-akt-till-mars-an
garagehang
solcellskollens-podcast
rss-laddstationen-med-elbilen-i-sverige
rss-snacka-om-ai