Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(541)

Unsupervised Learning: No. 181

Some absolutely fascinating research has just come out on what percentages and types of vulnerabilities are actually exploited in the wild. It found that only 5.5% of vulnerabilities discovered betwee...

11 Jun 201924min

Grit is the Ultimate Privilege

An argument that we should acknowledge grit as one of the most powerful causal factors in success, and figure out ways to bring its benefits to everyone.Become a Member: https://danielmiessler.com/upg...

8 Jun 20196min

Why Software Remains Insecure

A concise explanation of why software continues to have security and quality problems after decades of supposedly trying to address the problem.Become a Member: https://danielmiessler.com/upgradeSee o...

6 Jun 20194min

Unsupervised Learning: No. 179

The Deepfakes thing is already starting to have an impact, and it didn't even involve actual Deepfake (GAN ML) technology. A video was spread of Nancy Pelosi speaking very slowly and seeming to stumbl...

28 Mai 201917min

Unsupervised Learning: No. 178

Trump has semi-banned the use of foreign telecom gear, which is really a direct shot at Huawei and China. moreBaltimore’s IT systems are still being held hostage after 2 weeks. Of all the cities in th...

24 Mai 201923min

Unsupervised Learning: No. 177

My Takeaways from the 2019 DBIR Report My Summary The ReportThe DOJ has unsealed the indictment against those who they believe hacked Anthem in 2015, and they are Chinese Nationals. They didn't reveal...

14 Mai 201922min

Finding Clarity on the Exodus of the New Left

A short essay that attempts to wrap a simple narrative around what's happening with the exodus of the New Left, and what it's doing to the moderate left, center, and right that they left behind.Become...

4 Mai 201910min

Unsupervised Learning: No. 175

Deepfakes are about to seriously erode our collective ability to tell truth from fiction, and this is already a big enough problem without them. Think of every problem you care about, and realize this...

1 Mai 201936min